February 2021 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

RGW: Multiple Site does not sync olds data
by 特木勒 17 May '21

17 May '21

Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to enable full sync for bucket or this is a bug? Thanks

3 17

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

How's the maturity of CephFS and how's the maturity of Ceph erasure code?
by fanyuanli 31 Mar '21

31 Mar '21

Hi all, I'm a rookie in CEPH. I want to ask two questions. One is the maturity of cephfs, the file system of CEPH, and whether it is recommended for production environment. The other is the maturity of CEPH's erasure code and whether it can be used in production environment. Are the above two questions explained in the official documents? I may not see where they are. Thank you! Fred Fan

4 4

Fwd: ceph-fuse false passed X_OK check
by Alex Taylor 30 Mar '21

30 Mar '21

Hi Patrick, Any updates? Looking forward to your reply :D On Thu, Dec 17, 2020 at 11:39 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote: > > On Wed, Dec 16, 2020 at 5:46 PM Alex Taylor <alexu4993(a)gmail.com> wrote: > > > > Hi Cephers, > > > > I'm using VSCode remote development with a docker server. It worked OK > > but fails to start the debugger after /root mounted by ceph-fuse. The > > log shows that the binary passes access X_OK check but cannot be > > actually executed. see: > > > > ``` > > strace_log: access("/root/.vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7", > > X_OK) = 0 > > > > root@develop:~# ls -alh > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > -rw-r--r-- 1 root root 978 Dec 10 13:06 > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > ``` > > > > I also test the access syscall on ext4, xfs and even cephfs kernel > > client, all of them return -EACCES, which is expected (the extension > > will then explicitly call chmod +x). > > > > After some digging in the code, I found it is probably caused by > > https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L5549-L5550. > > So here come two questions: > > 1. Is this a bug or is there any concern I missed? > > I tried reproducing it with the master branch and could not. It might > be due to an older fuse/ceph. I suggest you upgrade! > I tried the master(332a188d9b3c4eb5c5ad2720b7299913c5a772ee) as well and the issue still exists. My test program is: ``` #include <stdio.h> #include <unistd.h> int main() { int r; const char path[] = "test"; r = access(path, F_OK); printf("file exists: %d\n", r); r = access(path, X_OK); printf("file executable: %d\n", r); return 0; } ``` And the test result: ``` # local filesystem: ext4 root@f626800a6e85:~# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:13 test root@f626800a6e85:~# ./a.out file exists: 0 file executable: -1 root@f626800a6e85:~# findmnt -t fuse.ceph-fuse TARGET SOURCE FSTYPE OPTIONS /root/mnt ceph-fuse fuse.ceph-fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other root@f626800a6e85:~# cd mnt # ceph-fuse root@f626800a6e85:~/mnt# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:10 test root@f626800a6e85:~/mnt# ./a.out file exists: 0 file executable: 0 root@f626800a6e85:~/mnt# ./test bash: ./test: Permission denied ``` Again, ceph-fuse says file `test` is executable but in fact it can't be executed. The kernel version I'm testing on is: ``` root@f626800a6e85:~/mnt# uname -ar Linux f626800a6e85 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux ``` Please try the program above and make sure you're running it as root user, thank you. And if the reproduction still fails, please let me know the kernel version. > > 2. It works again with fuse_default_permissions=true, any drawbacks if > > this option is set? > > Correctness (ironically, for you) and performance. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >

2 1

CephFS max_file_size
by Mark Schouten 25 Mar '21

25 Mar '21

Hi, There is a default limit of 1TiB for the max_file_size in CephFS. I altered that to 2TiB, but I now got a request for storing a file up to 7TiB. I'd expect the limit to be there for a reason, but what is the risk of setting that value to say 10TiB? -- Mark Schouten <mark(a)tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208

5 4

fixing future rctimes
by Dan van der Ster 23 Mar '21

23 Mar '21

Hi all, We have a few subdirs with an rctime in the future. # getfattr -n ceph.dir.rctime session # file: session ceph.dir.rctime="2576387188.090" I can't find any subdir or item in that directory with that rctime, so I presume that there was previously a file and that rctime cannot go backwards [1] Is there any way to fix these rctimes so they show the latest ctime of the subtree? Also -- are we still relying on the client clock to set the rctime / ctime of a file? Would it make sense to limit ctime/rctime for any update to the current time on the MDS ? Best Regards, Dan [1] https://github.com/ceph/ceph/pull/24023/commits/920ef964311a61fcc6c0d6671b7…

2 3

MON slow ops and growing MON store
by Janek Bevendorff 18 Mar '21

18 Mar '21

Hi, All of a sudden, we are experiencing very concerning MON behaviour. We have five MONs and all of them have thousands up to tens of thousands of slow ops, the oldest one blocking basically indefinitely (at least the timer keeps creeping up). Additionally, the MON stores keep inflating heavily. Under normal circumstances we have about 450-550MB there. Right now its 27GB and growing (rapidly). I tried restarting all MONs, I disabled auto-scaling (just in case) and checked the system load and hardware. I also restarted the MGR and MDS daemons, but to no avail. Is there any way I can debug this properly? I can’t seem to find how I can actually view what ops are causing this and what client (if any) may be responsible for it. Thanks Janek

4 13

Removing secondary data pool from mds
by Michael Thomas 13 Mar '21

13 Mar '21

I have a cephfs secondary (non-root) data pool with unfound and degraded objects that I have not been able to recover[1]. I created an additional data pool and used "setfattr -n ceph.dir.layout.pool' and a very long rsync to move the files off of the degraded pool and onto the new pool. This has completed, and using find + 'getfattr -n ceph.file.layout.pool', I verified that no files are using the old pool anymore. No ceph.dir.layout.pool attributes point to the old pool either. However, the old pool still reports that there are objects in the old pool, likely the same ones that were unfound/degraded from before: https://pastebin.com/qzVA7eZr Based on a old message from the mailing list[2], I checked the MDS for stray objects (ceph daemon mds.ceph4 dump cache file.txt ; grep -i stray file.txt) and found 36 stray entries in the cache: https://pastebin.com/MHkpw3DV. However, I'm not certain how to map these stray cache objects to clients that may be accessing them. 'rados -p fs.data.archive.frames ls' shows 145 objects. Looking at the parent of each object shows 2 strays: for obj in $(cat rados.ls.txt) ; do echo $obj ; rados -p fs.data.archive.frames getxattr $obj parent | strings ; done [...] 10000020fa1.00000000 10000020fa1 stray6 10000020fbc.00000000 10000020fbc stray6 [...] ...before getting stuck on one object for over 5 minutes (then I gave up): 1000005b1af.00000083 What can I do to make sure this pool is ready to be safely deleted from cephfs (ceph fs rm_data_pool archive fs.data.archive.frames)? --Mike [1]https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QHFOGEKXK7VDNNSKR74BA6IIMGGIXBXA/#7YQ6SSTESM5LTFVLQK3FSYFW5FDXJ5CF [2]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005233.h…

2 5

February 2021 Tech Talk and Code Walk-through
by Mike Perez 10 Mar '21

10 Mar '21

Hi everyone! I'm excited to announce two talks we have on the schedule for February 2021: Jason Dillaman will be giving part 2 to the librbd code walk-through. The stream starts on February 23rd at 18:00 UTC / 19:00 CET / 1:00 PM EST / 10:00 AM PST https://tracker.ceph.com/projects/ceph/wiki/Code_Walkthroughs Part 1: https://www.youtube.com/watch?v=L0x61HpREy4 -------------- What's New in the Pacific Release Hear Sage Weil give a live update on the development of the Pacific Release. The stream starts on February 25th at 17:00 UTC / 18:00 CET / 12 PM EST / 9 AM PST. https://ceph.io/ceph-tech-talks/ All live streams will be recorded and -- Mike Perez

1 2

Ceph User Survey Now Available
by Mike Perez 09 Mar '21

09 Mar '21

Hi everyone! Be sure to make your voice heard by taking the Ceph User Survey before April 2, 2021. This information will help guide the Ceph community’s investment in Ceph and the Ceph community's future development. https://ceph.io/user-survey/ Thank you to the Ceph User Survey Working Group for designing this year's survey! * Anantha Adiga * Anthony D'Atri * Paul Mezzanini * Stefan Kooman https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group We will provide the final results in the coming months after the survey has ended. -- Mike Perez

1 1

Unable to delete bucket - endless multipart uploads?
by David Monschein 08 Mar '21

08 Mar '21

Hi All, We've been dealing with what seems to be a pretty annoying bug for a while now. We are unable to delete a customer's bucket that seems to have an extremely large number of aborted multipart uploads. I've had $(radosgw-admin bucket rm --bucket=pusulax --purge-objects) running in a screen session for almost 3 weeks now and it's still not finished; it's most likely stuck in a loop or something. The screen session with debug-rgw=10 spams billions of these messages: 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/d3/04d33e18-3f13-433c-b924-56602d702d60-31.msg.2~0DTalUjTHsnIiKraN1klwIFO88Vc2E3.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/d7/04d7ad26-c8ec-4a39-9938-329acd6d9da7-102.msg.2~K_gAeTpfEongNvaOMNa0IFwSGPpQ1iA.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/da/04da4147-c949-4c3a-aca6-e63298f5ff62-102.msg.2~-hXBSFcjQKbMkiyEqSgLaXMm75qFzEp.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/db/04dbb0e6-dfb0-42fb-9d0f-49cceb18457f-102.msg.2~B5EhGgBU5U_U7EA5r8IhVpO3Aj2OvKg.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/df/04df39be-06ab-4c72-bc63-3fac1d2700a9-11.msg.2~_8h5fWlkNrIMqcrZgNbAoJfc8BN1Xx-.meta[] This is probably the 2nd or 3rd time I've been unable to delete this bucket. I also tried running $(radosgw-admin bucket check --fix --check-objects --bucket=pusulax) before kicking off the delete job, but that didn't work either. Here is the bucket in question, the num_objects counter never decreases after trying to delete the bucket: [root@os5 ~]# radosgw-admin bucket stats --bucket=pusulax { "bucket": "pusulax", "num_shards": 144, "tenant": "", "zonegroup": "dbb69c5b-b33f-4af2-950c-173d695a4d2c", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "c0d0b8a5-c63c-4c24-9dab-8deee88dbf0b.3209338.4", "marker": "c0d0b8a5-c63c-4c24-9dab-8deee88dbf0b.3292800.7", "index_type": "Normal", "owner": "REDACTED", "ver": "0#115613,1#115196,2#115884,3#115497,4#114649,5#114150,6#116127,7#114269,8#115220,9#115092,10#114003,11#114538,12#115235,13#113463,14#114928,15#115135,16#115535,17#114867,18#116010,19#115766,20#115274,21#114818,22#114805,23#114853,24#114099,25#114359,26#114966,27#115790,28#114572,29#114826,30#114767,31#115614,32#113995,33#115305,34#114227,35#114342,36#114144,37#114704,38#114088,39#114738,40#114133,41#114520,42#114420,43#114168,44#113820,45#115093,46#114788,47#115522,48#114713,49#115315,50#115055,51#114513,52#114086,53#114401,54#114079,55#113649,56#114089,57#114157,58#114064,59#115224,60#114753,61#114686,62#115169,63#114321,64#114949,65#115075,66#115003,67#114993,68#115320,69#114392,70#114893,71#114219,72#114190,73#114868,74#113432,75#114882,76#115300,77#114755,78#114598,79#114221,80#114895,81#114031,82#114566,83#113849,84#115155,85#113790,86#113334,87#113800,88#114856,89#114841,90#115073,91#113849,92#114554,93#114820,94#114256,95#113840,96#114838,97#113784,98#114876,99#115524,100#115686,101#112969,102#112156,103#112635,104#112732,105#112933,106#112412,107#113090,108#112239,109#112697,110#113444,111#111730,112#112446,113#114479,114#113318,115#113032,116#112048,117#112404,118#114545,119#112563,120#112341,121#112518,122#111719,123#112273,124#112014,125#112979,126#112209,127#112830,128#113186,129#112944,130#111991,131#112865,132#112688,133#113819,134#112586,135#113275,136#112172,137#113019,138#112872,139#113130,140#112716,141#112091,142#111859,143#112773", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0", "mtime": "2020-06-17 20:27:16.685833Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#", "usage": { "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 97009704, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 94737, "num_objects": 3592952 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } We're running 14.2.16 on our RGWs and OSD nodes. Anyone have any ideas? Is it possible to target this bucket via rados directly to try and delete it? I'm weary of doing stuff like that though. Thanks in advance. - Dave Monschein

3 2

"optimal" tunables on release upgrade
by Matthew Vernon 03 Mar '21

03 Mar '21

Hi, Having been slightly caught out by tunables on my Octopus upgrade[0], can I just check that if I do ceph osd crush tunables optimal That will update the tunables on the cluster to the current "optimal" values (and move a lot of data around), but that this doesn't mean they'll change next time I upgrade the cluster or anything like that? It's not quite clear from the documentation whether the next time "optimal" tunables change that'll be applied to a cluster where I've set tunables thus, or if tunables are only ever changed by a fresh invocation of ceph osd crush tunables... [I assume the same answer applies to "default"?] Regards, Matthew [0] I foolishly thought a cluster initially installed as Jewel would have jewel tunables -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

2 1

Remapped PGs
by David Orman 02 Mar '21

02 Mar '21

Hi, We see that we have 5 'remapped' PGs, but are unclear why/what to do about it. We shifted some target ratios for the autobalancer and it resulted in this state. When adjusting ratio, we noticed two OSDs go down, but we just restarted the container for those OSDs with podman, and they came back up. Here's status output: ################### root@ceph01:~# ceph status INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w) mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl, ceph02.gcglcg, ceph04.smbdew, ceph05.yropto osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs data: pools: 3 pools, 1057 pgs objects: 18.00M objects, 69 TiB usage: 119 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 1056 active+clean 1 active+clean+scrubbing+deep io: client: 859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr root@ceph01:~# ################### When I look at ceph pg dump, I don't see any marked as remapped: ################### root@ceph01:~# ceph pg dump |grep remapped INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 dumped all root@ceph01:~# ################### Any idea what might be going on/how to recover? All OSDs are up. Health is 'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman 2.0.3.

2 4

15.2.8 mgr keep crashing every few days
by levin ng 01 Mar '21

01 Mar '21

Hi all, I’d recently deployed ceph 15.2.8 with 3(mon,mgr,rgw,mds) and 4 (osd) total 7 host, however I encountered mgr crash a few times a week, the crashing mgr can be any one of 3. I couldn’t identify the problem behind and here is the crash info, appreciate anyone if you have suggestions that I could narrow it down. Thank you very much. { "assert_condition": "ret == 0", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc", "assert_func": "void Thread::create(const char*, size_t)", "assert_line": 157, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f833addc700 time 2021-02-10T20:00:32.980508+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc: 157: FAILED ceph_assert(ret == 0)\n", "assert_thread_name": "mgr-fin", "backtrace": [ "(()+0x12b20) [0x7f835a51cb20]", "(gsignal()+0x10f) [0x7f8358f6d7ff]", "(abort()+0x127) [0x7f8358f57c35]", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f835c07b735]", "(()+0x27a8fe) [0x7f835c07b8fe]", "(()+0x34cef6) [0x7f835c14def6]", "(DispatchQueue::start()+0x3a) [0x7f835c29697a]", "(AsyncMessenger::ready()+0xcd) [0x7f835c3340cd]", "(Messenger::add_dispatcher_head(Dispatcher*)+0x68) [0x7f835c3f8478]", "(MonClient::get_monmap_and_config()+0xbb) [0x7f835c3f66ab]", "(ceph_mount_info::init()+0x4d) [0x7f834298435d]", "(()+0x3680f) [0x7f8342cd280f]", "(()+0x19d421) [0x7f835ba5c421]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x1221d4) [0x7f835b9e11d4]", "(()+0x122c55) [0x7f835b9e1c55]", "(()+0x19cf27) [0x7f835ba5bf27]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(_PyFunction_FastCallDict()+0x122) [0x7f835b9b9ec2]", "(_PyObject_FastCallDict()+0x70e) [0x7f835b9bac9e]", "(()+0x10dc70) [0x7f835b9ccc70]", "(_PyObject_FastCallDict()+0x6ec) [0x7f835b9bac7c]", "(PyObject_CallFunctionObjArgs()+0xe8) [0x7f835b9dbd48]", "(_PyEval_EvalFrameDefault()+0x2588) [0x7f835ba5eef8]", "(()+0xf99b4) [0x7f835b9b89b4]", "(()+0x179e60) [0x7f835ba38e60]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x10d5) [0x7f835ba5da45]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0xfa326) [0x7f835b9b9326]", "(()+0x179e60) [0x7f835ba38e60]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(_PyFunction_FastCallDict()+0x122) [0x7f835b9b9ec2]", "(_PyObject_FastCallDict()+0x70e) [0x7f835b9bac9e]", "(()+0x10dc70) [0x7f835b9ccc70]", "(PyObject_Call()+0x4b) [0x7f835b9c1acb]", "(PyObject_CallMethod()+0x10b) [0x7f835ba5ac6b]", "(ActivePyModule::handle_command(ModuleCommand const&, MgrSession const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*)+0x222) [0x55bc0b8a0cb2]", "(()+0x1b0fdd) [0x55bc0b8f5fdd]", "(Context::complete(int)+0xd) [0x55bc0b8b0bdd]", "(Finisher::finisher_thread_entry()+0x1a5) [0x7f835c10b465]", "(()+0x814a) [0x7f835a51214a]", "(clone()+0x43) [0x7f8359032f23]" ], "ceph_version": "15.2.8", "crash_id": "2021-02-10T20:00:32.989661Z_201fd5fb-6e0a-4b50-8a95-fdf9ed9aeb81", "entity_name": "mgr.sds01-cp.cwcxek", "os_id": "centos", "os_name": "CentOS Linux", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "e1c15d685283e7598b128a37a328ba86ec433dfef97597ac9453b5d52608feda", "timestamp": "2021-02-10T20:00:32.989661Z", "utsname_hostname": "sds01-cp", "utsname_machine": "x86_64", "utsname_release": "4.18.0-240.10.1.el8_3.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Dec 16 03:30:52 EST 2020" }

4 5

Getting started with cephadm
by Peter Childs 28 Feb '21

28 Feb '21

I'm new to ceph, and I've been trying to set up a new cluster with 16 computers with 30 disks each and 6 SSD (plus boot disks), 256G of memory, IB Networking. (ok its currently 15 but never mind) When I take them over about 10 OSD's each they start having problems starting the OSD up and I can normally fix this by rebooting them and it will continue again for a while, and it is possible to get them up to the full complement with a bit of poking around. (Once its working it fne unless you start adding services or moving the OSD's around Is there anything I can change to make it a bit more stable. I've already set fs.aio-max-nr = 1048576 kernel.pid_max = 4194303 fs.file-max = 500000 which made it a bit better, but I feel it could be even better. I'm currently trying to upgrade to 15.2.9 from the default cephadm version of octopus. The upgrade is going very very slowly. I'm currently using podman if that helps, I'm not sure if docker would be better? (I've mainly used singularity when I've handled containers before) Thanks in advance Peter Childs

2 4

Slow cluster / misplaced objects - Ceph 15.2.9
by David Orman 27 Feb '21

27 Feb '21

Hi, We've got an interesting issue we're running into on Ceph 15.2.9. We're experiencing VERY slow performance from the cluster, and extremely slow misplaced object correction, with very little cpu/disk/network utilization (almost idle) across all nodes in the cluster. We have 7 servers in this cluster, 24 rotational OSDs, and two NVMEs with 12 OSD's worth of DB/WAL files on them. The OSDs are all equal weighted, so the tree is pretty straightforward: root@ceph01:~# ceph osd tree Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2149.39062 root default -2 2149.39062 rack rack1 -5 307.05579 host ceph01 0 hdd 12.79399 osd.0 up 1.00000 1.00000 1 hdd 12.79399 osd.1 up 1.00000 1.00000 2 hdd 12.79399 osd.2 up 1.00000 1.00000 3 hdd 12.79399 osd.3 up 1.00000 1.00000 4 hdd 12.79399 osd.4 up 1.00000 1.00000 5 hdd 12.79399 osd.5 up 1.00000 1.00000 6 hdd 12.79399 osd.6 up 1.00000 1.00000 7 hdd 12.79399 osd.7 up 1.00000 1.00000 8 hdd 12.79399 osd.8 up 1.00000 1.00000 9 hdd 12.79399 osd.9 up 1.00000 1.00000 10 hdd 12.79399 osd.10 up 1.00000 1.00000 11 hdd 12.79399 osd.11 up 1.00000 1.00000 12 hdd 12.79399 osd.12 up 1.00000 1.00000 13 hdd 12.79399 osd.13 up 1.00000 1.00000 14 hdd 12.79399 osd.14 up 1.00000 1.00000 15 hdd 12.79399 osd.15 up 1.00000 1.00000 16 hdd 12.79399 osd.16 up 1.00000 1.00000 17 hdd 12.79399 osd.17 up 1.00000 1.00000 18 hdd 12.79399 osd.18 up 1.00000 1.00000 19 hdd 12.79399 osd.19 up 1.00000 1.00000 20 hdd 12.79399 osd.20 up 1.00000 1.00000 21 hdd 12.79399 osd.21 up 1.00000 1.00000 22 hdd 12.79399 osd.22 up 1.00000 1.00000 23 hdd 12.79399 osd.23 up 1.00000 1.00000 -7 307.05579 host ceph02 24 hdd 12.79399 osd.24 up 1.00000 1.00000 25 hdd 12.79399 osd.25 up 1.00000 1.00000 26 hdd 12.79399 osd.26 up 1.00000 1.00000 27 hdd 12.79399 osd.27 up 1.00000 1.00000 28 hdd 12.79399 osd.28 up 1.00000 1.00000 29 hdd 12.79399 osd.29 up 1.00000 1.00000 30 hdd 12.79399 osd.30 up 1.00000 1.00000 31 hdd 12.79399 osd.31 up 1.00000 1.00000 32 hdd 12.79399 osd.32 up 1.00000 1.00000 33 hdd 12.79399 osd.33 up 1.00000 1.00000 34 hdd 12.79399 osd.34 up 1.00000 1.00000 35 hdd 12.79399 osd.35 up 1.00000 1.00000 36 hdd 12.79399 osd.36 up 1.00000 1.00000 37 hdd 12.79399 osd.37 up 1.00000 1.00000 38 hdd 12.79399 osd.38 up 1.00000 1.00000 39 hdd 12.79399 osd.39 up 1.00000 1.00000 40 hdd 12.79399 osd.40 up 1.00000 1.00000 41 hdd 12.79399 osd.41 up 1.00000 1.00000 42 hdd 12.79399 osd.42 up 1.00000 1.00000 43 hdd 12.79399 osd.43 up 1.00000 1.00000 44 hdd 12.79399 osd.44 up 1.00000 1.00000 45 hdd 12.79399 osd.45 up 1.00000 1.00000 46 hdd 12.79399 osd.46 up 1.00000 1.00000 47 hdd 12.79399 osd.47 up 1.00000 1.00000 -9 307.05579 host ceph03 48 hdd 12.79399 osd.48 up 1.00000 1.00000 49 hdd 12.79399 osd.49 up 1.00000 1.00000 50 hdd 12.79399 osd.50 up 1.00000 1.00000 51 hdd 12.79399 osd.51 up 1.00000 1.00000 52 hdd 12.79399 osd.52 up 1.00000 1.00000 53 hdd 12.79399 osd.53 up 1.00000 1.00000 54 hdd 12.79399 osd.54 up 1.00000 1.00000 55 hdd 12.79399 osd.55 up 1.00000 1.00000 56 hdd 12.79399 osd.56 up 1.00000 1.00000 57 hdd 12.79399 osd.57 up 1.00000 1.00000 58 hdd 12.79399 osd.58 up 1.00000 1.00000 59 hdd 12.79399 osd.59 up 1.00000 1.00000 60 hdd 12.79399 osd.60 up 1.00000 1.00000 61 hdd 12.79399 osd.61 up 1.00000 1.00000 62 hdd 12.79399 osd.62 up 1.00000 1.00000 63 hdd 12.79399 osd.63 up 1.00000 1.00000 64 hdd 12.79399 osd.64 up 1.00000 1.00000 65 hdd 12.79399 osd.65 up 1.00000 1.00000 66 hdd 12.79399 osd.66 up 1.00000 1.00000 67 hdd 12.79399 osd.67 up 1.00000 1.00000 68 hdd 12.79399 osd.68 up 1.00000 1.00000 69 hdd 12.79399 osd.69 up 1.00000 1.00000 70 hdd 12.79399 osd.70 up 1.00000 1.00000 71 hdd 12.79399 osd.71 up 1.00000 1.00000 -11 307.05579 host ceph04 72 hdd 12.79399 osd.72 up 1.00000 1.00000 73 hdd 12.79399 osd.73 up 1.00000 1.00000 74 hdd 12.79399 osd.74 up 1.00000 1.00000 75 hdd 12.79399 osd.75 up 1.00000 1.00000 76 hdd 12.79399 osd.76 up 1.00000 1.00000 77 hdd 12.79399 osd.77 up 1.00000 1.00000 78 hdd 12.79399 osd.78 up 1.00000 1.00000 79 hdd 12.79399 osd.79 up 1.00000 1.00000 80 hdd 12.79399 osd.80 up 1.00000 1.00000 81 hdd 12.79399 osd.81 up 1.00000 1.00000 82 hdd 12.79399 osd.82 up 1.00000 1.00000 83 hdd 12.79399 osd.83 up 1.00000 1.00000 84 hdd 12.79399 osd.84 up 1.00000 1.00000 85 hdd 12.79399 osd.85 up 1.00000 1.00000 86 hdd 12.79399 osd.86 up 1.00000 1.00000 87 hdd 12.79399 osd.87 up 1.00000 1.00000 88 hdd 12.79399 osd.88 up 1.00000 1.00000 89 hdd 12.79399 osd.89 up 1.00000 1.00000 90 hdd 12.79399 osd.90 up 1.00000 1.00000 91 hdd 12.79399 osd.91 up 1.00000 1.00000 92 hdd 12.79399 osd.92 up 1.00000 1.00000 93 hdd 12.79399 osd.93 up 1.00000 1.00000 94 hdd 12.79399 osd.94 up 1.00000 1.00000 95 hdd 12.79399 osd.95 up 1.00000 1.00000 -13 307.05579 host ceph05 96 hdd 12.79399 osd.96 up 1.00000 1.00000 97 hdd 12.79399 osd.97 up 1.00000 1.00000 98 hdd 12.79399 osd.98 up 1.00000 1.00000 99 hdd 12.79399 osd.99 up 1.00000 1.00000 100 hdd 12.79399 osd.100 up 1.00000 1.00000 101 hdd 12.79399 osd.101 up 1.00000 1.00000 102 hdd 12.79399 osd.102 up 1.00000 1.00000 103 hdd 12.79399 osd.103 up 1.00000 1.00000 104 hdd 12.79399 osd.104 up 1.00000 1.00000 105 hdd 12.79399 osd.105 up 1.00000 1.00000 106 hdd 12.79399 osd.106 up 1.00000 1.00000 107 hdd 12.79399 osd.107 up 1.00000 1.00000 108 hdd 12.79399 osd.108 up 1.00000 1.00000 109 hdd 12.79399 osd.109 up 1.00000 1.00000 110 hdd 12.79399 osd.110 up 1.00000 1.00000 111 hdd 12.79399 osd.111 up 1.00000 1.00000 112 hdd 12.79399 osd.112 up 1.00000 1.00000 113 hdd 12.79399 osd.113 up 1.00000 1.00000 114 hdd 12.79399 osd.114 up 1.00000 1.00000 115 hdd 12.79399 osd.115 up 1.00000 1.00000 116 hdd 12.79399 osd.116 up 1.00000 1.00000 117 hdd 12.79399 osd.117 up 1.00000 1.00000 118 hdd 12.79399 osd.118 up 1.00000 1.00000 119 hdd 12.79399 osd.119 up 1.00000 1.00000 -15 307.05579 host ceph06 120 hdd 12.79399 osd.120 up 1.00000 1.00000 121 hdd 12.79399 osd.121 up 1.00000 1.00000 122 hdd 12.79399 osd.122 up 1.00000 1.00000 123 hdd 12.79399 osd.123 up 1.00000 1.00000 124 hdd 12.79399 osd.124 up 1.00000 1.00000 125 hdd 12.79399 osd.125 up 1.00000 1.00000 126 hdd 12.79399 osd.126 up 1.00000 1.00000 127 hdd 12.79399 osd.127 up 1.00000 1.00000 128 hdd 12.79399 osd.128 up 1.00000 1.00000 129 hdd 12.79399 osd.129 up 1.00000 1.00000 130 hdd 12.79399 osd.130 up 1.00000 1.00000 131 hdd 12.79399 osd.131 up 1.00000 1.00000 132 hdd 12.79399 osd.132 up 1.00000 1.00000 133 hdd 12.79399 osd.133 up 1.00000 1.00000 134 hdd 12.79399 osd.134 up 1.00000 1.00000 135 hdd 12.79399 osd.135 up 1.00000 1.00000 136 hdd 12.79399 osd.136 up 1.00000 1.00000 137 hdd 12.79399 osd.137 up 1.00000 1.00000 138 hdd 12.79399 osd.138 up 1.00000 1.00000 139 hdd 12.79399 osd.139 up 1.00000 1.00000 140 hdd 12.79399 osd.140 up 1.00000 1.00000 141 hdd 12.79399 osd.141 up 1.00000 1.00000 142 hdd 12.79399 osd.142 up 1.00000 1.00000 143 hdd 12.79399 osd.143 up 1.00000 1.00000 -17 307.05579 host ceph07 144 hdd 12.79399 osd.144 up 1.00000 1.00000 145 hdd 12.79399 osd.145 up 1.00000 1.00000 146 hdd 12.79399 osd.146 up 1.00000 1.00000 147 hdd 12.79399 osd.147 up 1.00000 1.00000 148 hdd 12.79399 osd.148 up 1.00000 1.00000 149 hdd 12.79399 osd.149 up 1.00000 1.00000 150 hdd 12.79399 osd.150 up 1.00000 1.00000 151 hdd 12.79399 osd.151 up 1.00000 1.00000 152 hdd 12.79399 osd.152 up 1.00000 1.00000 153 hdd 12.79399 osd.153 up 1.00000 1.00000 154 hdd 12.79399 osd.154 up 1.00000 1.00000 155 hdd 12.79399 osd.155 up 1.00000 1.00000 156 hdd 12.79399 osd.156 up 1.00000 1.00000 157 hdd 12.79399 osd.157 up 1.00000 1.00000 158 hdd 12.79399 osd.158 up 1.00000 1.00000 159 hdd 12.79399 osd.159 up 1.00000 1.00000 160 hdd 12.79399 osd.160 up 1.00000 1.00000 161 hdd 12.79399 osd.161 up 1.00000 1.00000 162 hdd 12.79399 osd.162 up 1.00000 1.00000 163 hdd 12.79399 osd.163 up 1.00000 1.00000 164 hdd 12.79399 osd.164 up 1.00000 1.00000 165 hdd 12.79399 osd.165 up 1.00000 1.00000 166 hdd 12.79399 osd.166 up 1.00000 1.00000 167 hdd 12.79399 osd.167 up 1.00000 1.00000 root@ceph01:~# The data distribution looks relatively even (we use upmap as the balancer): root@ceph01:~# ceph osd df Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.0 MiB 14 GiB 7.1 TiB 44.73 0.93 35 up 1 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.8 MiB 13 GiB 7.2 TiB 43.46 0.90 33 up 2 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.4 MiB 16 GiB 6.2 TiB 51.27 1.07 39 up 3 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 4.9 MiB 15 GiB 6.7 TiB 47.38 0.99 36 up 4 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 812 KiB 13 GiB 7.2 TiB 43.37 0.90 35 up 5 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 31 KiB 16 GiB 6.2 TiB 51.27 1.07 40 up 6 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 506 KiB 13 GiB 7.3 TiB 43.32 0.90 34 up 7 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 904 KiB 15 GiB 6.4 TiB 49.93 1.04 38 up 8 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.6 MiB 14 GiB 6.7 TiB 47.35 0.98 37 up 9 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.5 MiB 16 GiB 6.1 TiB 52.58 1.09 40 up 10 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.6 TiB 75 MiB 16 GiB 6.1 TiB 52.42 1.09 41 up 11 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.4 MiB 13 GiB 7.2 TiB 43.36 0.90 34 up 12 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 586 KiB 13 GiB 7.3 TiB 43.31 0.90 33 up 13 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 222 KiB 14 GiB 7.1 TiB 44.71 0.93 35 up 14 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16 GiB 6.1 TiB 52.58 1.09 41 up 15 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.2 MiB 14 GiB 7.1 TiB 44.73 0.93 34 up 16 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16 GiB 6.1 TiB 52.53 1.09 40 up 17 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.3 MiB 15 GiB 6.4 TiB 49.92 1.04 38 up 18 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16 GiB 6.1 TiB 52.61 1.09 45 up 19 hdd 12.79399 1.00000 13 TiB 5.4 TiB 5.3 TiB 4.0 MiB 13 GiB 7.4 TiB 42.15 0.88 34 up 20 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 705 KiB 16 GiB 6.1 TiB 52.53 1.09 40 up 21 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.5 MiB 16 GiB 6.1 TiB 52.58 1.09 40 up 22 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.7 MiB 16 GiB 6.1 TiB 52.59 1.09 40 up 23 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.9 MiB 16 GiB 6.1 TiB 52.58 1.09 41 up 24 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14 GiB 6.7 TiB 47.35 0.98 36 up 25 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.1 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 26 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 592 KiB 14 GiB 7.1 TiB 44.75 0.93 34 up 27 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.8 MiB 14 GiB 6.9 TiB 46.07 0.96 36 up 28 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.4 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 29 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 340 KiB 15 GiB 6.4 TiB 49.92 1.04 39 up 30 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 93 KiB 14 GiB 6.9 TiB 46.03 0.96 35 up 31 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.1 MiB 13 GiB 7.2 TiB 43.57 0.91 35 up 32 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 5.0 MiB 15 GiB 6.6 TiB 48.68 1.01 38 up 33 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 175 KiB 13 GiB 7.2 TiB 43.44 0.90 33 up 34 hdd 12.79399 1.00000 13 TiB 6.9 TiB 6.8 TiB 77 MiB 16 GiB 5.9 TiB 53.86 1.12 43 up 35 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 2.3 MiB 15 GiB 6.6 TiB 48.68 1.01 37 up 36 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.2 MiB 16 GiB 6.2 TiB 51.25 1.07 41 up 37 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 5.0 MiB 14 GiB 6.7 TiB 47.37 0.99 36 up 38 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.7 MiB 13 GiB 7.2 TiB 43.44 0.90 35 up 39 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 341 KiB 15 GiB 6.7 TiB 47.38 0.99 37 up 40 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.1 MiB 14 GiB 7.1 TiB 44.73 0.93 34 up 41 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 22 KiB 16 GiB 6.1 TiB 52.59 1.09 40 up 42 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 1.9 MiB 14 GiB 6.7 TiB 47.33 0.98 36 up 43 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 485 KiB 16 GiB 6.1 TiB 52.59 1.09 40 up 44 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.1 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 45 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.3 MiB 16 GiB 6.1 TiB 52.53 1.09 40 up 46 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 124 KiB 14 GiB 6.9 TiB 46.08 0.96 37 up 47 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.2 MiB 13 GiB 7.2 TiB 43.37 0.90 33 up 48 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 141 KiB 16 GiB 6.1 TiB 52.58 1.09 40 up 49 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.9 MiB 13 GiB 7.2 TiB 43.64 0.91 33 up 50 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 745 KiB 16 GiB 6.1 TiB 52.52 1.09 41 up 51 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 186 KiB 16 GiB 6.1 TiB 52.55 1.09 41 up 52 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.0 MiB 16 GiB 6.2 TiB 51.24 1.07 41 up 53 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 737 KiB 14 GiB 6.7 TiB 47.33 0.98 38 up 54 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.3 MiB 14 GiB 7.1 TiB 44.66 0.93 34 up 55 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 97 KiB 14 GiB 7.1 TiB 44.73 0.93 34 up 56 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.2 MiB 14 GiB 7.1 TiB 44.76 0.93 35 up 57 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.5 MiB 14 GiB 7.1 TiB 44.78 0.93 35 up 58 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 80 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 59 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 4.5 MiB 14 GiB 6.9 TiB 46.08 0.96 36 up 60 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 903 KiB 14 GiB 7.1 TiB 44.73 0.93 34 up 61 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1 KiB 14 GiB 7.1 TiB 44.65 0.93 34 up 62 hdd 12.79399 1.00000 13 TiB 6.9 TiB 6.8 TiB 425 KiB 16 GiB 5.9 TiB 53.87 1.12 41 up 63 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.1 MiB 14 GiB 7.2 TiB 43.56 0.91 34 up 64 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 578 KiB 16 GiB 6.2 TiB 51.27 1.07 39 up 65 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.6 MiB 13 GiB 7.2 TiB 43.44 0.90 33 up 66 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 288 KiB 13 GiB 7.2 TiB 43.40 0.90 34 up 67 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 413 KiB 14 GiB 6.7 TiB 47.34 0.98 37 up 68 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.3 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 69 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.0 MiB 13 GiB 7.2 TiB 43.44 0.90 33 up 70 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.2 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 71 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.3 MiB 14 GiB 7.2 TiB 43.36 0.90 35 up 72 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.7 MiB 16 GiB 6.2 TiB 51.29 1.07 39 up 73 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14 GiB 7.1 TiB 44.73 0.93 34 up 74 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.6 TiB 1.2 MiB 14 GiB 7.1 TiB 44.64 0.93 34 up 75 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 1.5 MiB 15 GiB 6.4 TiB 49.95 1.04 38 up 76 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 3.8 MiB 15 GiB 6.6 TiB 48.68 1.01 37 up 77 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.0 MiB 14 GiB 7.1 TiB 44.65 0.93 34 up 78 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14 GiB 7.1 TiB 44.68 0.93 35 up 79 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 984 KiB 14 GiB 6.7 TiB 47.36 0.98 36 up 80 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.6 MiB 13 GiB 7.2 TiB 43.45 0.90 34 up 81 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.2 MiB 16 GiB 6.2 TiB 51.26 1.07 41 up 82 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 4.9 MiB 15 GiB 6.6 TiB 48.61 1.01 37 up 83 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 827 KiB 14 GiB 6.7 TiB 47.33 0.98 36 up 84 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 2.7 MiB 16 GiB 6.2 TiB 51.22 1.07 39 up 85 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 868 KiB 15 GiB 6.4 TiB 49.94 1.04 40 up 86 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.1 MiB 16 GiB 6.1 TiB 52.57 1.09 42 up 87 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 702 KiB 16 GiB 6.1 TiB 52.54 1.09 41 up 88 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 12 KiB 14 GiB 7.1 TiB 44.76 0.93 35 up 89 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.9 MiB 14 GiB 6.9 TiB 46.07 0.96 35 up 90 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.8 MiB 13 GiB 7.2 TiB 43.48 0.90 34 up 91 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 836 KiB 16 GiB 6.1 TiB 52.55 1.09 40 up 92 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.0 MiB 13 GiB 7.2 TiB 43.43 0.90 33 up 93 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.5 MiB 16 GiB 6.1 TiB 52.57 1.09 41 up 94 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 459 KiB 13 GiB 7.2 TiB 43.44 0.90 34 up 95 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.1 MiB 16 GiB 6.2 TiB 51.28 1.07 39 up 96 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.1 MiB 14 GiB 7.1 TiB 44.75 0.93 35 up 97 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 662 KiB 14 GiB 7.1 TiB 44.75 0.93 34 up 98 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.9 MiB 14 GiB 6.9 TiB 45.97 0.96 35 up 99 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.8 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 100 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14 GiB 7.1 TiB 44.76 0.93 36 up 101 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.6 MiB 15 GiB 6.4 TiB 49.95 1.04 40 up 102 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.9 MiB 14 GiB 7.1 TiB 44.65 0.93 34 up 103 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 643 KiB 16 GiB 6.1 TiB 52.57 1.09 41 up 104 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 5.5 MiB 15 GiB 6.4 TiB 49.98 1.04 38 up 105 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 830 KiB 13 GiB 7.2 TiB 43.46 0.90 33 up 106 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 1.0 MiB 14 GiB 6.7 TiB 47.34 0.98 36 up 107 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 286 KiB 16 GiB 6.2 TiB 51.23 1.07 39 up 108 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 961 KiB 15 GiB 6.6 TiB 48.68 1.01 37 up 109 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 291 KiB 16 GiB 6.1 TiB 52.56 1.09 42 up 110 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1.0 MiB 14 GiB 6.9 TiB 46.02 0.96 36 up 111 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 2.7 MiB 16 GiB 6.2 TiB 51.22 1.07 39 up 112 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 631 KiB 15 GiB 6.6 TiB 48.67 1.01 37 up 113 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.5 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 114 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.0 MiB 15 GiB 6.4 TiB 49.94 1.04 39 up 115 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1.5 MiB 14 GiB 6.9 TiB 46.05 0.96 35 up 116 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.1 MiB 15 GiB 6.4 TiB 49.93 1.04 38 up 117 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 194 KiB 13 GiB 7.2 TiB 43.44 0.90 35 up 118 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.1 MiB 15 GiB 6.4 TiB 49.96 1.04 38 up 119 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.9 MiB 14 GiB 7.2 TiB 43.46 0.90 33 up 120 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.8 MiB 16 GiB 6.1 TiB 52.55 1.09 40 up 121 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.7 MiB 16 GiB 6.1 TiB 52.54 1.09 40 up 122 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 2.4 MiB 15 GiB 6.4 TiB 49.97 1.04 38 up 123 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1003 KiB 14 GiB 6.9 TiB 46.02 0.96 35 up 124 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 526 KiB 14 GiB 6.7 TiB 47.35 0.98 36 up 125 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 92 KiB 15 GiB 6.7 TiB 47.37 0.98 37 up 126 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.9 MiB 14 GiB 7.1 TiB 44.74 0.93 34 up 127 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.5 MiB 16 GiB 6.2 TiB 51.26 1.07 39 up 128 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.4 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 129 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 3.6 MiB 15 GiB 6.6 TiB 48.66 1.01 38 up 130 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.8 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 131 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.1 MiB 16 GiB 6.2 TiB 51.28 1.07 40 up 132 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.7 MiB 16 GiB 6.2 TiB 51.25 1.07 39 up 133 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 230 KiB 16 GiB 6.2 TiB 51.25 1.07 39 up 134 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.0 MiB 14 GiB 7.2 TiB 43.45 0.90 34 up 135 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.4 MiB 13 GiB 7.2 TiB 43.46 0.90 33 up 136 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.6 MiB 16 GiB 6.2 TiB 51.29 1.07 39 up 137 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.6 MiB 14 GiB 7.1 TiB 44.75 0.93 36 up 138 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.9 MiB 15 GiB 6.4 TiB 49.98 1.04 40 up 139 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 21 KiB 16 GiB 6.2 TiB 51.23 1.07 39 up 140 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.4 MiB 14 GiB 7.1 TiB 44.74 0.93 36 up 141 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 359 KiB 13 GiB 7.2 TiB 43.42 0.90 34 up 142 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 27 KiB 13 GiB 7.2 TiB 43.46 0.90 33 up 143 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 395 KiB 14 GiB 6.9 TiB 46.07 0.96 35 up 144 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.2 MiB 14 GiB 7.1 TiB 44.75 0.93 34 up 145 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.3 MiB 14 GiB 7.1 TiB 44.66 0.93 35 up 146 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 4.0 MiB 14 GiB 6.9 TiB 46.06 0.96 35 up 147 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 46 KiB 16 GiB 6.1 TiB 52.55 1.09 40 up 148 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.8 MiB 16 GiB 6.2 TiB 51.23 1.07 39 up 149 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.4 MiB 14 GiB 7.1 TiB 44.70 0.93 35 up 150 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.4 MiB 14 GiB 7.1 TiB 44.65 0.93 35 up 151 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.7 MiB 14 GiB 7.1 TiB 44.65 0.93 35 up 152 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.3 MiB 15 GiB 6.2 TiB 51.26 1.07 39 up 153 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 2.3 MiB 15 GiB 6.4 TiB 49.94 1.04 39 up 154 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14 GiB 6.7 TiB 47.37 0.99 36 up 155 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14 GiB 6.7 TiB 47.35 0.98 38 up 156 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.0 MiB 16 GiB 6.2 TiB 51.26 1.07 41 up 157 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.8 MiB 14 GiB 6.9 TiB 46.07 0.96 36 up 158 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 3.0 MiB 16 GiB 6.2 TiB 51.25 1.07 40 up 159 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.9 MiB 15 GiB 6.4 TiB 49.93 1.04 38 up 160 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.4 MiB 14 GiB 6.9 TiB 46.03 0.96 35 up 161 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.1 MiB 16 GiB 6.2 TiB 51.25 1.07 39 up 162 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.6 TiB 4.1 MiB 13 GiB 7.1 TiB 44.64 0.93 34 up 163 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 445 KiB 14 GiB 7.1 TiB 44.69 0.93 34 up 164 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.9 MiB 15 GiB 6.4 TiB 49.97 1.04 38 up 165 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.5 MiB 16 GiB 6.2 TiB 51.25 1.07 40 up 166 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.5 MiB 13 GiB 7.2 TiB 43.45 0.90 33 up 167 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.0 MiB 15 GiB 6.4 TiB 49.98 1.04 38 up TOTAL 2.1 PiB 1.0 PiB 1023 TiB 630 MiB 2.4 TiB 1.1 PiB 48.09 MIN/MAX VAR: 0.88/1.12 STDDEV: 3.50 root@ceph01:~# The cluster status, however, shows we have many misplaced objects and a few remapped PGs. We had been performing maintenance (upgrade to Ceph 15.2.9), and initially the misplaced object count was dropping rapidly at a rate of about 400-500MB/s and 100-200 objects/s. However, once it hit a certain point, it dropped to 0-4MB/s and 0 objects/s (it's not 0, it is progressing, but it's below 1 object/s): root@ceph01:~# ceph -s Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_WARN 5 slow ops, oldest one blocked for 84 sec, daemons [osd.141,osd.46] have slow ops. services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 25h) mgr: ceph05.yropto(active, since 11h), standbys: ceph01.aqkgbl, ceph03.ytkuyr, ceph04.smbdew, ceph02.ndynmo osd: 168 osds: 168 up (since 9h), 168 in (since 6M); 10 remapped pgs task status: data: pools: 3 pools, 1057 pgs objects: 166.90M objects, 632 TiB usage: 1.0 PiB used, 1.1 PiB / 2.1 PiB avail pgs: 583177/1001370417 objects misplaced (0.058%) 1039 active+clean 11 active+clean+scrubbing 5 active+remapped+backfilling 1 active+clean+scrubbing+deep 1 active+clean+laggy io: client: 4.5 MiB/s rd, 31 MiB/s wr, 75 op/s rd, 59 op/s wr recovery: 2.7 MiB/s, 0 objects/s root@ceph01:~# root@ceph01:~# ceph pg dump |grep remapped Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 2.3d1 163805 0 0 0 0 682273001472 0 0 7462 7462 active+*remapped*+backfilling 2021-02-26T01:28:06.212904+0000 53782'2773494 53782:5593409 [48,122,99,92,158,31] 48 [48,122,99,92,158,34] 48 50907'2762333 2021-02-23T20:13:50.501757+0000 50907'2762333 2021-02-23T20:13:50.501757+0000 0 2.341 163180 0 0 136844 0 679698268160 0 0 8569 8569 active+*remapped*+backfilling 2021-02-26T01:27:30.693520+0000 53782'2694466 53782:5497907 [65,125,79,147,45,100] 65 [65,18,79,147,45,100] 65 51073'2688628 2021-02-24T09:38:41.588350+0000 51073'2688628 2021-02-24T09:38:41.588350+0000 0 2.1e1 162402 0 0 150624 0 676505321472 0 0 7893 7893 active+*remapped*+backfilling 2021-02-25T15:36:29.121352+0000 53782'3105605 53782:5885140 [112,63,126,0,147,25] 112 [112,50,126,0,147,25] 112 50907'3086340 2021-02-22T23:14:49.510535+0000 47263'2883486 2021-02-01T15:43:32.089129+0000 0 2.268 163459 0 0 136459 0 680743612416 0 0 7337 7337 active+*remapped*+backfilling 2021-02-25T15:27:58.957497+0000 53782'2713107 53782:5536140 [93,98,150,49,26,135] 93 [93,98,150,62,26,135] 93 50907'2698319 2021-02-23T14:16:11.171370+0000 50907'2698319 2021-02-23T14:16:11.171370+0000 0 2.285 163380 0 0 158866 0 680507670528 0 0 8148 8148 active+*remapped*+backfilling 2021-02-26T01:26:31.442018+0000 53782'3191803 53782:5986426 [67,166,37,90,114,138] 67 [67,147,37,90,114,138] 67 51073'3177251 2021-02-23T23:51:51.751625+0000 47597'2917541 2021-02-03T20:14:38.762384+0000 0 dumped all root@ceph01:~# Even if we stop the scrubs/deep scrubs, things don't improve re: recovery rate or cluster performance. We've monitored the HDDs/NVME backing these OSDs and with the scrubs stopped, they are nearly idle. CPU is almost idle. There's nothing to indicate any problem at all with the underlying hardware that we can find, at least from a load/failure perspective. Also, interestingly, rados bench on a newly created **replicated** pool is also extremely slow, as in < 50MB/s. Crush rule for our EC (4+2) pool, but we see the performance issues on even a newly created triple-replicated pool: root@ceph01:~# ceph osd crush rule dump ecpool Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 { "rule_id": 1, "rule_name": "ecpool", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } Cluster/pool space: root@ceph01:~# ceph df Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 2.1 PiB 1.1 PiB 1.0 PiB 1.0 PiB 48.09 TOTAL 2.1 PiB 1.1 PiB 1.0 PiB 1.0 PiB 48.09 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 78 MiB 182 233 MiB 0 295 TiB ecpool 2 1024 632 TiB 166.90M 1023 TiB 53.65 589 TiB rbd 3 32 59 MiB 5 178 MiB 0 295 TiB root@ceph01:~# We're at a loss for why this cluster has suddenly started behaving this way. With the disks (all of them, on all nodes) effectively being idle with all the scrubs disabled, plenty of CPU available, and no network issues (iperf on all paths is full line rate - ~10GB/s - we have a dedicated public and cluster network), it's befuddling. We appreciate any assistance you can provide, and are happy to provide any information that might be useful.

4 6

MDSs report damaged metadata
by ricardo.re.azevedo＠gmail.com 26 Feb '21

26 Feb '21

Hi all, My cephfs MDS is reporting damaged metadata following the addition (and remapping) of 12 new OSDs. `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type "backtrace" which is very concerning. ` ceph tell mds.database-0 scrub start / recursive repair` seems to have no effect on the damage. What does this sort of damage mean? Is there anything I can do to recover these files? > ceph status reports: cluster: id: 692905c0-f271-4cd8-9e43-1c32ef8abd13 health: HEALTH_ERR 1 MDSs report damaged metadata 630 pgs not deep-scrubbed in time 630 pgs not scrubbed in time services: mon: 3 daemons, quorum database-0,file-server,webhost (age 37m) mgr: webhost(active, since 3d), standbys: file-server, database-0 mds: cephfs:1 {0=database-0=up:active} 2 up:standby osd: 48 osds: 48 up (since 56m), 48 in (since 13d); 10 remapped pgs task status: scrub status: mds.database-0: idle data: pools: 7 pools, 633 pgs objects: 60.82M objects, 231 TiB usage: 336 TiB used, 246 TiB / 582 TiB avail pgs: 623 active+clean 6 active+remapped+backfilling 4 active+remapped+backfill_wait Thanks for the help. Best, Ricardo

2 2

Nautilus Cluster Struggling to Come Back Online
by William Law 26 Feb '21

26 Feb '21

I guess as a sort of follow up from my previous post. Our Nautilus (14.2.16 on ubuntu 18.04) cluster had some sort of event that caused many of the machines to have memory errors. The aftermath is that initially some OSDs had (and continue to have) this error https://tracker.ceph.com/issues/48827 others won't start for various reasons. The OSDs that *will* start are badly behind the current epoch for the most part. It sounds very similar to this: https://blog.noc.grnet.gr/2016/10/18/surviving-a-ceph-cluster-outage-the-ha… We are having trouble getting things back online. I think the path forward is to: -set noup/nodown/noout/nobackfill/and wait for the OSDs that run to come up; we were making good progress yesterday until some of the OSDs crashed with OOM errors. We are again moving forward but understandably nervous. -export the PGs from questionable OSDs and and then rebuild the OSDs; import the PGs if necessary (very likely). Repeat until we are up. Any suggestions for increasing speed? We are using noup/nobackfill/norebalance/pause but the epoch catchup is taking a very long time. Any tips for keeping the epoch from moving forward or speeding up the OSDs catching up? How can we estimate how long it should take? Thank you for any ideas or assistance anyone can provide. Will

2 2

ceph version of new daemons deployed with orchestrator
by Kenneth Waegeman 26 Feb '21

26 Feb '21

Hi all, I am running a cluster managed by orchestrator/cephadm. I installed new host for OSDS yesterday, the osd daemons were automatically created using drivegroups service specs (https://docs.ceph.com/en/latest/cephadm/drivegroups/#drivegroups <https://docs.ceph.com/en/latest/cephadm/drivegroups/#drivegroups>) and they started with a 15.2.9 image, instead of 15.2.8 which all daemons of the cluster are running. I did not yet run ceph orch upgrade to 15.2.9. Is there a way to lock the version of OSDS/daemons created by orchestrator/cephadm? Thanks! Kenneth

2 2

Newbie Help With ceph-mgr
by duluxoz 26 Feb '21

26 Feb '21

Hi All, My ceph-mgr keeps stopping (for some unknown reason) after about an hour or so (but has run for up to 2-3 hours before stopping). Up till now I've simple restarted it with 'ceph-mgr -i ceph01'. Is this normal behaviour, or if it isn't, what should I be looking for in the logs? I was thinking of writing a quick cron script (with 'ceph-mgr -i ceph01') to run on the hour every hour to restart it, but figured that there had to be a better way - especially if ceph-mgr was crashing instead of being a "feature". Any ideas/advice? Thanks in advance Dulux-Oz -- Peregrine IT Signature *Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P When you want it done /right/ ‒ the first time! Phone: +61 4 0411 0089 Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net> View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack> This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege. If you have received this Email in error, please notify the author and delete this message immediately.

1 0

Newbie Requesting Help - Please, This Is Driving Me Mad/Crazy!
by matthew＠peregrineit.net 26 Feb '21

26 Feb '21

Hi Everyone, Let me apologise upfront: If this isn't the correct List to post to If this has been answered already (& I've missed it in my searching) If this has ended up double posted If I've in any way given (or about to give) offence to anyone I really need some help. I'm trying to get a simple single host Pilot/Test Cluster up and running. I'm using CentOS 8 (fully updated), and Ceph-Octopus (latest version from the Ceph Repo). I have both ceph-mon and ceph-mgr working/running (although ceph-mge keeps stopping/crashing after about 1-3 hours or so - but that's another issue), and my first osd (and only osd at this point) *appears* to be working, but when I issue the command 'systemctl start ceph-osd@0' the ceph-osd daemon won't spin up and thus when I issue 'ceph -s' the result says the 'osd: 1 osds: 0 up, 0 in'. I've gone through the relevant logs but I can't seem to find the issue. I'm doing this as a Manual Install because I want to actually *learn* what's going on during the install/etc. I know I can use cephadmin (in a production environment), but as I said, I'm trying to learn how everything "fits together". I've read and re-read the official Ceph Documentation and followed the following steps/commands to get Ceph installed and running: Ran the following commands: su - useradd -d /home/ceph -m ceph -p <password> mkdir /home/ceph/.ssh Added a public SSH Key to /home/ceph/.ssh/authorized_keys. Ran the following commands: chmod 600 /home/ceph/.ssh/* chown ceph:ceph -R /home/ceph/.ssh Added the ceph.repo details to /etc/yum.repos.d/ceph.repo (as per the Ceph Documentation). Ran the following command: dnf -y install qemu-kvm qemu-guest-agent libvirt gdisk ceph Created the /etc/ceph/ceph.conf file (see listing below). Ran the following commands: ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r' ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/keyring chown -R ceph:ceph /etc/ceph/ chown -R ceph:ceph /var/lib/ceph/ monmaptool --create --add ceph01 192.168.0.10 --fsid 98e84f97-031f-4958-bd54-22305f6bc738 /etc/ceph/monmap mkdir /var/lib/ceph/mon/ceph-ceph01 chown -R ceph:ceph /var/lib/ceph sudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring firewall-cmd --add-service=http --permanent firewall-cmd --add-service=ceph --permanent firewall-cmd --add-service=ceph-mon --permanent firewall-cmd --reload chmod -R 750 /var/lib/ceph/ systemctl start ceph-mon@ceph01 ceph mon enable-msgr2 mkdir /var/lib/ceph/mgr/ceph-ceph01 chown ceph:ceph /var/lib/ceph/mgr/ceph-ceph01 ceph auth get-or-create mgr.ceph01 mon 'allow profile mgr' mds 'allow *' osd 'allow *' -o /var/lib/ceph/mgr/ceph-ceph01/keyring ceph-mgr -i ceph01 Fdisked 3 hdds (sdb, sdc, sdd) as GPT partitions. Ran the following commands: mkfs.xfs /dev/sdb1 mkfs.xfs /dev/sdc1 mkfs.xfs /dev/sdd1 mkdir -p /var/lib/ceph/osd/ceph-{0,1,2} chown -R ceph:ceph /var/lib/ceph/osd mount /dev/sdb1 /var/lib/ceph/osd/ceph-0 mount /dev/sdc1 /var/lib/ceph/osd/ceph-1 mount /dev/sdd1 /var/lib/ceph/osd/ceph-2 So, at this point everything is working, although 'ceph -s' does give a Health Warning about not having the required number of osds (as per the /etc/ceph/ceph.conf file). Here is what I did to create and (fail to) run my first osd (osd.0): Ran the following commands: sudo -u ceph ceph osd new $(uuidgen) sudo -u ceph ceph auth get-or-create osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' -o /var/lib/ceph/osd/ceph-0/keyring sudo -u ceph ceph-osd -i 0 --mkfs ceph osd crush add 0 2 host=ceph01 systemctl start ceph-osd@0 The osd shows up when I issue the command 'ceph osd ls'. The key shows up when I issue the command 'ceph auth ls'. But as I said above, when I issue the command 'ceph -s' it shows 'osd: 1 osds: 0 up, 0 in'. And when I look at the systemctl status for ceph-osd@0 it simply said it failed with 'exit code'. The /etc/ceph/ceph.conf listing: [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx fsid = 98e84f97-031f-4958-bd54-22305f6bc738 mon_host = ceph01 public_network = 192.168.0.0/24 [mgr] mgr_initial_modules = dashboard alerts balancer restful status [mgr.ceph01] log_file = /var/log/ceph/ceph-mgr.ceph01.log [mon] mon_initial_members = ceph01 mon_data_size_warn = 8589934592 mon_allow_pool_delete = true [mon.ceph01] host = ceph01 mon_addr = 192.168.0.10 log_file = /var/log/ceph/ceph-mon.ceph01.log [osd] allow_ec_overwrites = true osd_crush_chooseleaf_type = 1 osd_journal_size = 10240 osd_pool_default_min_size = 2 osd_pool_default_pg_num = 128 osd_pool_default_pgp_num = 128 osd_pool_default_size = 3 osd_scrub_auto_repair = true osd_scrub_begin_hour = 3 osd_scrub_end_hour = 11 pg_autoscale_mode = on [osd.0] host = ceph01 log_file = /var/log/ceph/ceph-osd.0.log [osd.1] host = ceph01 log_file = /var/log/ceph/ceph-osd.1.log [osd.2] host = ceph01 log_file = /var/log/ceph/ceph-osd.2.log So, could someone please point out to me where I'm going wrong - I know its got to be something super-simple, but this has been driving me mad for over a week now. Thanks in advance Dulux-Oz

4 6

Re: Question about per MDS journals
by John Spray 25 Feb '21

25 Feb '21

osdc/Journaler is for RDB, client/Journaler is for CephFS. On Thu, Feb 25, 2021 at 8:26 AM 조규진 <bori19960(a)snu.ac.kr> wrote: > > Hi, John. > > Thanks for your kind reply! > > While i'm checking the code that you recommend to check and other .cc files about journal, I find that there is two Journaler class. > One is at "src/osdc/Journaler.h" and the other one is at "src/journal/Journaler.h". > If you don't mind, could you tell me which one is for MDS journal? and the differences between them? > > Thanks. > kyujin > > 2021년 2월 25일 (목) 오전 1:15, John Spray <jcspray(a)gmail.com>님이 작성: >> >> On Wed, Feb 24, 2021 at 9:10 AM 조규진 <bori19960(a)snu.ac.kr> wrote: >> > >> > Hi. >> > >> > I'm a newbie in CephFS and I have some questions about how per-MDS journals >> > work. >> > In Sage's paper (osdi '06), I read that each MDSs has its own journal and >> > it lazily flushes metadata modifications on OSD cluster. >> > What I'm wondering is that some directory operations like rename work with >> > multiple metadata and It may work on two or more MDSs and their journals, >> > so I think it needs some mechanisms to construct a transaction that works >> > on multiple journals like some distributed transaction mechanisms. >> > >> > Could anybody explains how per-MDS journals work in such directory >> > operations? or recommends some references about it? >> >> Your intuition is correct: these transactions span multiple MDS journals. >> >> The code for this stuff is somewhat long, in src/mds/Server.cc, but >> here are a couple of pointers if you're interested in untangling it: >> - Server::handle_client_rename is the entry point >> - The MDS which handles the client request sends MMDSPeerRequest >> messages to peers in rename_prepare_witness, and waits for >> acknowledgements before writing EUpdate events to its journal >> - The peer(s) write EPeerUpdate(OP_PREPARE) events to their journals >> during prepare, and EPeerUpdate(OP_COMMIT) after the first MDS has >> completed. >> >> John >> >> >> >> > >> > Thanks. >> > kyujin. >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Erasure coded calculation
by Simon Sutter 25 Feb '21

25 Feb '21

Hello everyone! I'm trying to calculate the theoretical usable storage of a ceph cluster with erasure coded pools. I have 8 nodes and the profile for all data pools will be k=6 m=2. If every node has 6 x 1TB wouldn't the calculation be like this: RAW capacity: 8Nodes x 6Disks x 1TB = 48TB Loss to m=2: 48TB / 8Nodes x 2m = 12TB EC capacity: 48TB - 12TB = 36TB At the moment I have one cluster with 8 nodes and different disks than the sample (but every node has the same amount of disks and the same sized disks). The output of ceph df detail is: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 TOTAL 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 --- POOLS --- POOL ID PGS STORED OBJECTS %USED MAX AVAIL device_health_metrics 1 1 51 MiB 48 0 30 TiB rep_data_fs 2 32 14 KiB 3.41k 0 30 TiB rep_meta_fs 3 32 227 MiB 1.72k 0 30 TiB ec_bkp1 4 32 4.2 TiB 1.10M 6.11 67 TiB So ec_bkp1 uses 4.2TiB an there are 67TiB free usable Storage. This means total EC usable storage would be 71.2TiB. But calculating with the 109TiB RAW storage, shouldn't it be 81.75? Are the 10TiB just some overhead (that would be much overhead) or is the calculation not correct? And what If I want to expand the cluster in the first sample above by three nodes with 6 x 2TB, which means not the same sized disks as the others. Will the calculation with the same EC profile still be the same? RAW capacity: 8Nodes x 6Disks x 1TB + 3Nodes x 6Disks x 2TB = 84TB Loss to m=2: 84TB / 11Nodes x 2m = 15.27TB EC capacity: 84TB - 15.27TB = 68.72TB Thanks in advance, Simon

3 2

ceph slow at 80% full, mds nodes lots of unused memory
by Simon Oosthoek 25 Feb '21

25 Feb '21

Hi we've been running our Ceph cluster for nearly 2 years now (Nautilus) and recently, due to a temporary situation the cluster is at 80% full. We are only using CephFS on the cluster. Normally, I realize we should be adding OSD nodes, but this is a temporary situation, and I expect the cluster to go to <60% full quite soon. Anyway, we are noticing some really problematic slowdowns. There are some things that could be related but we are unsure... - Our 2 MDS nodes (1 active, 1 standby) are configured with 128GB RAM, but are not using more than 2GB, this looks either very inefficient, or wrong ;-) "ceph config dump |grep mds": mds basic mds_cache_memory_limit 107374182400 mds advanced mds_max_scrub_ops_in_progress 10 Perhaps we require more or different settings to properly use the MDS memory? - On all our OSD nodes, the memory line is red in "atop", though no swap is in use, it seems the memory on the OSD nodes is taking quite a beating, is this normal, or can we tweak settings to make it less stressed? This is the first time we are having performance issues like this, I think, I'd like to learn some commands to help me analyse this... I hope this will ring a bell with someone... Cheers /Simon

3 5

List number of buckets owned per user
by Marcelo 25 Feb '21

25 Feb '21

Hello. I'm trying to list the number of buckets that users have for monitoring purposes, but I need to list and count the number of buckets per user. Is it possible to get this information somewhere else? Thanks, Marcelo

3 2

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

Default data pool in CEPH
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi, I have a CEPH 15.2.4 running in a docker. How to configure for use a specific data pool? i try put the follow line in the ceph.conf but the changes not working. . [client.myclient] rbd default data pool = Mydatapool I need it to configure for erasure pool with cloudstack Can help me? , where is the ceph conf we i need configure? Thanks. -- Untitled Document

4 6

Re: NFS Ganesha NFSv3
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi Thanks for the reply. cephadm runs ceph containers automatically. How to set privileged mode in ceph container? -- > El 23/9/20 a las 13:24, Daniel Gryniewicz escribió: >> NFSv3 needs privileges to connect to the portmapper. Try running >> your docker container in privileged mode, and see if that helps. >> >> Daniel >> >> On 9/23/20 11:42 AM, Gabriel Medve wrote: >>> Hi, >>> >>> I have a CEPH 15.2.5 running in a docker , i configure nfs ganesha >>> with nfs version 3 but i can not mount it. >>> If configure ganesha with nfs version 4 i can mounted without >>> problems but i need the version 3 . >>> >>> The error is mount.nfs: Protocol not supported >>> >>> Can help me? >>> >>> Thanks. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- > Untitled Document

2 1

v15.2.9 Octopus released
by David Galloway 24 Feb '21

24 Feb '21

We're happy to announce the 9th backport release in the Octopus series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-9-Octopus-released Notable Changes --------------- * MGR: progress module can now be turned on/off, using the commands: ``ceph progress on`` and ``ceph progress off``. * OSD: PG removal has been optimized in this release. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.9.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 357616cbf726abb779ca75a551e8d02568e15b17

1 0

kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay)
by Frank Schilder 24 Feb '21

24 Feb '21

Hi all, I get these log messages all the time, sometimes also directly to the terminal: kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay) The cluster is healthy and the MDS complaining is actually both, configured and running as a standby-replay daemon. These messages show up at least every hour, but sometimes with much higher frequency. The cluster seems healthy though. A google search did not bring up anything useful. Can anyone shed some light on what this message means? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 3

Question about per MDS journals
by 조규진 24 Feb '21

24 Feb '21

Hi. I'm a newbie in CephFS and I have some questions about how per-MDS journals work. In Sage's paper (osdi '06), I read that each MDSs has its own journal and it lazily flushes metadata modifications on OSD cluster. What I'm wondering is that some directory operations like rename work with multiple metadata and It may work on two or more MDSs and their journals, so I think it needs some mechanisms to construct a transaction that works on multiple journals like some distributed transaction mechanisms. Could anybody explains how per-MDS journals work in such directory operations? or recommends some references about it? Thanks. kyujin.

2 1

multiple-domain for S3 on rgws with same ceph backend on one zone
by Simon Pierre DESROSIERS 24 Feb '21

24 Feb '21

Hello, We have functional ceph swarm with a pair of S3 rgw in front that uses A.B.C.D domain to be accessed. Now a new client asks to have access using the domain : E.C.D, but to already existing buckets. This is not a scenario discussed in the docs. Apparently, looking at the code and by trying it, rgw does not support multiple domains for the variable rgw_dns_name. But reading through parts of the code, I am no dev, and my c++ is 25 years rusty, I get the impression that maybe we could just add a second pair of rgw S3 servers that would give service to the same buckets, but using a different domain. Am I wrong ? Let's say this works, is this an unconscious behaviour that the ceph team would remove down the road ? Is there another solution that I might have missed ? We do not have multi-zone and there are no plans for it. And Cname (rgw_resolve_cname) seems to only be of use when using static sites (again, from my poor code reading abilities). Thank you -- **AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et peuvent contenir de l’information privilégiée ou confidentielle. Si vous avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que les pièces jointes, le cas échéant. La copie ou la redistribution non autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne peut être interprété qu’en conformité avec les lois et règlements qui régissent les pouvoirs des diverses instances décisionnelles compétentes de la Ville de Montréal.

4 7

Question about per-MDS journals
by bori19960＠naver.com 24 Feb '21

24 Feb '21

Hi. I'm a newbie in CephFS and I have some questions about how per-MDS journals work. In Sage's paper (osdi '06), I read that each MDSs has its own journal and it lazily flushes metadata modifications on OSD cluster. What I'm wondering is that some directory operations like rename work with multiple metadata and It may work on two or more MDSs and their journals, so I think it needs some mechanisms to construct a transaction that works on multiple journals like some distributed transaction mechanisms. Could anybody explains how per-MDS journals work in such directory operations? or recommends some references about it? Thanks. kyujin.

1 0

Question on multi-site
by Cary FitzHugh 24 Feb '21

24 Feb '21

Hello - We're using libRADOS directly for our communication between services. Some of the features are faster and more featured for our use cases than an S3 gateway. But we do want to leverage the ES Metadata search. It appears that the Metadata search is built on the object gateway. Question is - do files which are written directly to an OSD get replicated using the gateway, or is it only files which are written through the gateway that get replicated? Thanks. Cary

2 1

Storing 20 billions of immutable objects in Ceph, 75% <16KB
by Loïc Dachary 24 Feb '21

24 Feb '21

Bonjour, TL;DR: Is it more advisable to work on Ceph internals to make it friendly to this particular workload or write something similar to EOS[0] (i.e Rocksdb + Xrootd + RBD)? This is a followup of two previous mails[1] sent while researching this topic. In a nutshell, the Software Heritage project[1] currently has ~750TB and 10 billions objects, 75% of which have a size smaller than 16KB and 50% have a size smaller than 4KB. But they only account for ~5% of the 750TB: 25% of the objects have a size > 16KB and total ~700TB. The objects can be compressed by ~50% and 750TB only needs 350TB of actual storage. (if you're interested in the details see [2]). Let say those 10 billions objects are stored in a single 4+2 erasure coded pool with bluestore compression set for objects that have a size > 32KB and the smallest allocation size for bluestore set to 4KB[3]. The 750TB won't use the expected 350TB but about 30% more, i.e. ~450TB (see [4] for the maths). This space amplification is because storing a 1 byte object uses the same space as storing a 16KB object (see [5] to repeat the experience at home). In a 4+2 erasure coded pool, each of the 6 chunks will use no less than 4KB because that's the smallest allocation size for bluestore. That's 4 * 4KB = 16KB even when all that is needed is 1 byte. It was suggested[6] to have two different pools: one with a 4+2 erasure pool and compression for all objects with a size > 32KB that are expected to compress to 16KB. And another with 3 replicas for the smaller objects to reduce space amplification to a minimum without compromising on durability. A client looking for the object could make two simultaneous requests to the two pools. They would get 404 from one of them and the object from the other. Another workaround, is best described in the "Finding a needle in Haystack: Facebook’s photo storage"[9] paper and essentially boils down to using a database to store a map between the object name and its location. That does not scale out (writing the database index is the bottleneck) but it's simple enough and is successfully implemented in EOS[0] with >200PB worth of data and in seaweedfs[10], another promising object store software based on the same idea. Instead of working around the problem, maybe Ceph could be modified to make better use of the immutability of these objects[7], a hint that is apparently only used to figure out how to best compress it and for checksum calculation[8]. I honestly have not clue how difficult it would be. All I know is that it's not easy otherwise it would have been done already: there seem to be a general need for efficiently (space wise and performance wise) storing large quantities of objects smaller than 4KB. Is it more advisable to: * work on Ceph internals to make it friendly to this particular workload or, * write another implementation of "Finding a needle in Haystack: Facebook’s photo storage"[9] based on RBD[11]? I'm currently leaning toward working on Ceph internals but there are pros and cons to both approaches[12]. And since all this is still very new to me, there also is the possibility that I'm missing something. Maybe it's *super* difficult to improve Ceph in this way. I should try to figure that out sooner rather than later. I realize it's a lot to take in and unless you're facing the exact same problem there is very little chance you read that far :-) But if you did... I'm *really* interested to hear what yout think. In any case I'll report back to this thread once a decision has been made. Cheers [0] https://eos-web.web.cern.ch/eos-web/ [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF… https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX… [2] https://forge.softwareheritage.org/T3054 [3] https://github.com/ceph/ceph/blob/3f5e778ad6f055296022e8edabf701b6958fb602/… [4] https://forge.softwareheritage.org/T3052#58864 [5] https://forge.softwareheritage.org/T3052#58917 [6] https://forge.softwareheritage.org/T3052#58876 [7] https://docs.ceph.com/en/latest/rados/api/librados/#c.@3.LIBRADOS_ALLOC_HIN… [8] https://forge.softwareheritage.org/T3055 [9] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf [10] https://github.com/chrislusf/seaweedfs/wiki/Components [11] https://forge.softwareheritage.org/T3049 [12] https://forge.softwareheritage.org/T3054#58977 -- Loïc Dachary, Artisan Logiciel Libre

7 14

Multisite sync shards cleanup
by Szabo, Istvan (Agoda) 24 Feb '21

24 Feb '21

Hi, Is there a way to cleanup the sync shards and start from scratch? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

ceph-radosgw: Initialization timeout, failed to initialize
by Mathew Snyder 23 Feb '21

23 Feb '21

We have a Red Hat installation of Luminuous (full packages version: 12.2.8-128.1). We're experiencing an issue where the ceph-radosgw service will timeout during initialization and cycle through attempts every five minutes until it seems to just give up. Every other ceph service starts successfully. I tried looking at the health of the cluster, but anytime I run a command, whether ceph or radosgw-admin just to see a list of users, it seems to time out as well. I've used strace when attempting to start radosgw directly and was presented with a missing keyring error. I would be inclined to think that might be the problem, but wouldn't that also impact all of the other services? I haven't been able to find anything in the logs that would lead me down any paths. Everything I've looked at (journalctl, /var/log/messages, /var/log/ceph/ceph-rgw-server.log) all just say the same thing: the service attempted to start, it failed to initialize, entered a failed state, service stopped. This repeats. I only get pulled in to look at Ceph every now and then so I don't have enough knowledge to know how the various components interact or if anything external is having an impact. Is there anywhere that might be holding a bit of information? Thank you, Mathew Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 2

Possible to disable check: x pool(s) have no replicas configured
by Marc Roos 23 Feb '21

23 Feb '21

Is it possible to disable checking on 'x pool(s) have no replicas configured', so I don't have this HEALTH_WARN constantly. Or is there some other disadvantage of keeping some empty 1x replication test pools?

2 1

splitting Volume Group with odd number of PE in 2 logical volumes
by Gheorghiță Butnaru 23 Feb '21

23 Feb '21

Hello, Recently I deployed a small ceph cluster using cephadm. In this cluster, I have 3 OSD nodes with 8 HDDs Hitachi (9.1 TiB), 4 NVMes Micron_9300 (2.9 TiB), and 2 NVMes Intel Optane P4800X (375 GiB). I want to use spinning disks for the data block, 2.9 NVMes for the block.DB and the intel Optane for the block.wal. I tried with a spec file and also via the ceph dashboard but I encountered one problem. I would expect 1 lv on every data disk, 4 lv on wal disks, and 2 lv on DB disks. The problem arises on DB disks where only 1 lv gets created. After some debugging, I think that the problem is generated when the VG gets divided into 2. I have 763089 Total PE and the first LV was created using 381545 PE (round-up for 763089/2). Thanks to that, the creation of the second LV fails: Volume group "ceph-c7078851-d3c1-4745-96b6-f98a45d3da93" has insufficient free space (381544 extents): 381545 required. Is this an expected behavior or not? Should I create the LVs by myself? Gheorghita BUTNARU, Gheorghe Asachi Technical University of Iasi

1 0

Re: Network design issues
by Frank Schilder 23 Feb '21

23 Feb '21

Hi Stefan, OK, I added the ceph-users again :) Thanks for your reply, this is a lot of useful pointers. Yes, its Dell EMC switches running OS9 and I believe they support per-VLAN bandwidth reservations. It would be the easiest to configure and test. At the moment, I always see the slow ping times on both, the front- and back interface at the same time on exactly the same OSD pairs. If I reserve bandwidth to the replication VLAN and the slow ping times on the back interface disappear, this would be a really strong clue. I will go through everything after the weekend. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan(a)bit.nl> Sent: 12 February 2021 18:18 To: Frank Schilder Subject: Re: [ceph-users] Network design issues On 2/12/21 5:27 PM, Frank Schilder wrote: > Hi Stefan, > > do you want to keep this out of the ceph-users list or was it a click-and-miss? ^^ This, I recently switched to Thunderbird because of mail migration (from Mutt) ... and I'm not used to it yet. I *tried* to reply to all (incl. list) but might have screwed up. I would consider this as of general interest. > > Thanks for your detailed reply. I take it that I need to provide more info and will try to make a few sketches of the architecture. I think it will help explaining the problem. Some quick replies: > >> I'm curious what you changed. Want to share it? > > # ceph config set mds mds_max_caps_per_client 65536 > > Thread "cephfs: massive drop in MDS requests per second with increasing number of caps" > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/B7K6B5VXM3I… Ah yes, I've read that thread. Interesting. I haven't tested it out yet, but will do so. > > There are a number of config values with significantly too large defaults, this is one of them. Another one is mon_sync_max_payload_size. quite a few people have run into issues with that settig. We haven't had any issues with it yet. But perhaps I should downscale it as well. > >> Do you know what causes the slow ops? > > I don't care about slow ops under high load, these are to be expected. I worry about "slow ping times". These are not expected and are almost certainly caused by congestion of a link. Yeah sure, I would suspect that as well. Or "discards" from a switch because of errors, but those are less likely. > >> I don't quite get the 10G bottleneck. Sure, a client can saturate a 10 >> Gb/s link, but how does this affect storage <-> storage (replication) >> traffic and / or other clients? > > Because it all happens on the same physical link. We don't have a dedicated replication network. Its all mixed on the same hardware. If a 10G link is saturated, nothing moves any more through this particular link and the clients are so superior in capacity that they can easily starve parts of the internal ceph traffic in this way. > > Basically, we started out with a dedicated replication VLAN and decided to merge this with the access VLAN for simplicity of the set-up. Our networking is currently equivalent to having a single network only. Here the interfaces: > > ceph0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 > > ceph0.81@ceph0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 > > ceph0.82@ceph0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 > > $ ethtool ceph0 > Settings for ceph0: > Supported ports: [ ] > Supported link modes: Not reported > Supported pause frame use: No > Supports auto-negotiation: No > Supported FEC modes: Not reported > Advertised link modes: Not reported > Advertised pause frame use: No > Advertised auto-negotiation: No > Advertised FEC modes: Not reported > Speed: 60000Mb/s > Duplex: Full > Port: Other > PHYAD: 0 > Transceiver: internal > Auto-negotiation: off > Cannot get wake-on-lan settings: Operation not permitted > Link detected: yes > > the bond is 6x10G active-active, VLAN 81 is the access VLAN and 82 is the replication network. Goes all over the same lines. This config is very convenient for maintenance, but seems to suffer from not physically reserving bandwidth to VLAN 82. Maybe such a bandwidth reservation QOS definition could already help? Are these Dell / EMC switches? You might be able to give priority on a VLAN level, or "shape" bandwith based on VLANs. I know that Aristas (that we use) have support for that in newish firmware. You might also want to support "pause" frames (ethernet flow control) as that might help during congestion (a back off protocol), see: https://en.wikipedia.org/wiki/Ethernet_flow_control Just to note: we don't have a separate replication network / interfaces. We only have one network. Wido (den Hollander) and myself as well, don't see any added benefit of a separate network. You only waste bandwith if you split them up. And make debugging more complex in certain failure scenarios. Do you know what hashing is in use for the LACP port-channel? You want to use mac, ip and port (5-tuple). We use OpenvSwitch (OVS) a lot, and with OVS you can balance the load between the LACP links (by default it evaluates every 10 seconds if it should move flows around). I doubt there is a silver bullet, but hey, you never know. Do change one thing at a time, otherwise it will be hard to know what the effect is of each of the changes (they might even cancel each other out). > > I will provide a sketch of the set-up, I think this will make things more clear. I don't think we have an aggregated bandwidth problem, I believe what we have is a load distribution/priority problem over physical link members in the aggregation group "ceph0" on the storage servers. Yes, your issue makes more sense to me now. Do you have any metrics from the load on the individual links? Even bmon might be a useful tool. You might want to capture metrics (like every second or so) to detect "bursts" of traffic that might cause issues. Just to make sure you are on the right track. We use telegraf as metric collecting agent sending them to influxdb, but there are many more options. And then there are also other things to tune: tcp checksum offload et al. You might also hit IRQ balance issue, and there are also ways to overcome those. Are those single CPU systems? And / or AMD? NUMA might be a thing as well, and ideally you have the Ceph OSD daemons pinned to the CPU with the network / storage adapters connected. Finally, this might be of use: http://www.brendangregg.com/usemethod.html ;-). Gr. Stefan

2 5

Ceph nvme timeout and then aborting
by zxcs 23 Feb '21

23 Feb '21

Hi, I have one ceph cluster with nautilus 14.2.10 and one node has 3 SSD and 4 HDD each. Also has two nvmes as cache. (Means nvme0n1 cache for 0-2 SSD and Nvme1n1 cache for 3-7 HDD) but there is one nodes’ nvme0n1 always hit below issues(see name..I/O…timeout, aborting), and sudden this nvme0n1 disappear . After that i need reboot this node to recover. Any one hit same issue ? and how to slow it? Any suggestion are welcome. Thanks in advance! I am once googled the issue, and see below link, but not see any help https://askubuntu.com/questions/981657/cannot-suspend-with-nvme-m-2-ssd <https://askubuntu.com/questions/981657/cannot-suspend-with-nvme-m-2-ssd> From syslog Feb 19 01:31:52 ip kernel: [1275313.393211] nvme 0000:03:00.0: I/O 949 QID 12 timeout, aborting Feb 19 01:31:53 ip kernel: [1275314.389232] nvme 0000:03:00.0: I/O 728 QID 5 timeout, aborting Feb 19 01:31:53 ip kernel: [1275314.389247] nvme 0000:03:00.0: I/O 515 QID 7 timeout, aborting Feb 19 01:31:53 ip kernel: [1275314.389252] nvme 0000:03:00.0: I/O 516 QID 7 timeout, aborting Feb 19 01:31:53 ip kernel: [1275314.389257] nvme 0000:03:00.0: I/O 517 QID 7 timeout, aborting Feb 19 01:31:53 ip kernel: [1275314.389263] nvme 0000:03:00.0: I/O 82 QID 9 timeout, aborting Feb 19 01:31:53 ip kernel: [1275314.389271] nvme 0000:03:00.0: I/O 853 QID 13 timeout, aborting Feb 19 01:31:53 ip kernel: [1275314.389275] nvme 0000:03:00.0: I/O 854 QID 13 timeout, aborting Feb 19 01:32:23 ip kernel: [1275344.401708] nvme 0000:03:00.0: I/O 728 QID 5 timeout, reset controller Feb 19 01:32:52 ip kernel: [1275373.394112] nvme 0000:03:00.0: I/O 0 QID 0 timeout, reset controller Feb 19 01:33:53 ip ceph-osd[3179]: /build/ceph-14.2.10/src/common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, const char*, ceph::time_detail::coarse_mono_clock::rep)' thread 7f36c03fb700 time 2021-02-19 01:33:53.436018 Feb 19 01:33:53 ip ceph-osd[3179]: /build/ceph-14.2.10/src/common/HeartbeatMap.cc: 82: ceph_abort_msg("hit suicide timeout") Feb 19 01:33:53 ip ceph-osd[3179]: ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) Feb 19 01:33:53 ip ceph-osd[3179]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xdf) [0x83eb8c] Feb 19 01:33:53 ip ceph-osd[3179]: 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, unsigned long)+0x4a5) [0xec56f5] Feb 19 01:33:53 ip ceph-osd[3179]: 3: (ceph::HeartbeatMap::is_healthy()+0x106) [0xec6846] Feb 19 01:33:53 ip ceph-osd[3179]: 4: (OSD::handle_osd_ping(MOSDPing*)+0x67c) [0x8aaf0c] Feb 19 01:33:53 ip ceph-osd[3179]: 5: (OSD::heartbeat_dispatch(Message*)+0x1eb) [0x8b3f4b] Feb 19 01:33:53 ip ceph-osd[3179]: 6: (DispatchQueue::fast_dispatch(boost::intrusive_ptr<Message> const&)+0x27d) [0x12456bd] Feb 19 01:33:53 ip ceph-osd[3179]: 7: (ProtocolV2::handle_message()+0x9d6) [0x129b4e6] Feb 19 01:33:53 ip ceph-osd[3179]: 8: (ProtocolV2::handle_read_frame_dispatch()+0x160) [0x12ad330] Feb 19 01:33:53 ip ceph-osd[3179]: 9: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)+0x178) [0x12ad598] Feb 19 01:33:53 ip ceph-osd[3179]: 10: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x34) [0x12956b4] Feb 19 01:33:53 ip ceph-osd[3179]: 11: (AsyncConnection::process()+0x186) [0x126f446] Feb 19 01:33:53 ip ceph-osd[3179]: 12: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x7cd) [0x10b14cd] Feb 19 01:33:53 ip ceph-osd[3179]: 13: /usr/bin/ceph-osd() [0x10b3fd8] Feb 19 01:33:53 ip ceph-osd[3179]: 14: /usr/bin/ceph-osd() [0x162b59f] Feb 19 01:33:53 ip ceph-osd[3179]: 15: (()+0x76ba) [0x7f36c2ed46ba] Feb 19 01:33:53 ip ceph-osd[3179]: 16: (clone()+0x6d) [0x7f36c24db4dd] Feb 19 01:33:53 ip ceph-osd[3179]: *** Caught signal (Aborted) **

4 15

10G stackabe lacp switches
by mj 22 Feb '21

22 Feb '21

Hi, Hapy to report that we recently upgraded our three-host 24 OSD cluster from HDD filestore to SSD BlueStore. After a few months of use, their WEAR is still at 1%, and the cluster performance ("rados bench" etc) has dramatically improved. So all in all: yes, we're happy Samsung PM883 ceph users. :-) We currently have a "meshed" ceph setup, with the three hosts connected directly to each other over 10G ethernet, as described here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_2_.28… As we would like to be able to add more storage hosts, we need to loose the meshed network setup. My idea is to add two stacked 10G ethernet switches to the setup, so we can start using lacp bonded networking over two physical switches. Looking around, we can get refurb Cisco Small Business 550X for around 1300 euro. We also noticed that mikrotik and TP-Link have some even nicer-priced 10G switches, but those all lack bonding. :-( Therfore I'm asking here: anyone here with suggestions on what to look at, for nice-priced 10G stackable switches..? We would like to continue using ethernet, as we use that everywhere, and also performance-wise we're happy with what we currently have. Last december I wrote to mikrotik support, asking if they will support stacking / LACP any time soon, and their answer was: probably 2nd half of 2021. So, anyone here with interesting insights to share for ceph 10G ethernet storage networking? Thanks, MJ

12 17

Data Missing with RBD-Mirror
by Vikas Rana 22 Feb '21

22 Feb '21

Hi Friends, We have a very weird issue with rbd-mirror replication. As per the command output, we are in sync but the OSD usage on DR side doesn't match the Prod Side. On Prod, we are using close to 52TB but on DR side we are only 22TB. We took a snap on Prod and mounted the snap on DR side and compared the data and we found lot of missing data. Please see the output below. Please help us resolve this issue or point us in right direction. Thanks, -Vikas DR# rbd --cluster cephdr mirror pool status cifs --verbose health: OK images: 1 total 1 replaying research_data: global_id: 69656449-61b8-446e-8b1e-6cf9bd57d94a state: up+replaying description: replaying, master_position=[object_number=390133, tag_tid=4, entry_tid=447832541], mirror_position=[object_number=390133, tag_tid=4, entry_tid=447832541], entries_behind_master=0 last_update: 2021-01-29 15:10:13 DR# ceph osd pool ls detail pool 5 'cifs' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 1294 flags hashpspool stripe_width 0 application rbd removed_snaps [1~5] PROD# ceph df detail POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED cifs 17 N/A N/A 26.0TiB 30.10 60.4TiB 6860550 6.86M 873MiB 509MiB 52.1TiB DR# ceph df detail POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED cifs 5 N/A N/A 11.4TiB 15.78 60.9TiB 3043260 3.04M 2.65MiB 431MiB 22.8TiB PROD#:/vol/research_data# du -sh * 11T Flab1 346G KLab 1.5T More 4.4T ReLabs 4.0T WLab DR#:/vol/research_data# du -sh * 2.6T Flab1 14G KLab 52K More 8.0K RLabs 202M WLab

3 8

Outreachy May 2021
by Mike Perez 22 Feb '21

22 Feb '21

Hi everyone, I want to invite you to apply to an internship program called Outreachy! Outreachy provides three-month internships to work in Free and Open Source Software (FOSS). Outreachy internship projects may include programming, user experience, documentation, illustration, graphical design, or data science. Interns often find employment after their internship with Outreachy sponsors or jobs that use the skills they learned during their internship. Ceph has had ten projects submitted in Outreachy since 2018. Now we can submit more projects for our May-August 2021 round! The project can be coordinated on the etherpad: https://pad.ceph.com/p/project-ideas Projects need to be submitted by the mentor here for approval: https://www.outreachy.org/communities/cfp/ceph/ Outreachy internships run twice a year. The internships run from May to August and December to March. Interns are paid a stipend of $6,000 USD for the three months of work. Outreachy internships are entirely remote and are open to applicants around the world. Interns work remotely with experienced mentors. We expressly invite women (both cis and trans), trans men, and genderqueer people to apply. We also expressly invite applications from residents and nationals of the United States of any gender. They are Black/African American, Hispanic/Latin@, Native American/American Indian, Alaska Native, Native Hawaiian, or Pacific Islander. Anyone who faces under-representation, systematic bias, or discrimination in their country's technology industry is invited to apply. More details and eligibility criteria can be found here: https://www.outreachy.org/apply/eligibility/ The next Outreachy internship round is from May 24, 2021, until Aug. 24, 2021. Initial applications are currently open. Initial applications are due on Feb. 22, 2021, at 4 pm UTC. Apply today: https://www.outreachy.org/apply/ Applying to Outreachy is a little different than other internship programs. You'll fill out an initial application. If your initial application is approved, you'll move onto the five-week contribution phase. During the contribution phase, you'll make contact with project mentors and contribute to the project. Outreachy organizers have found that the most vital applicants contact mentors early, ask many questions, and continually submit contributions throughout the contribution phase. Please let Ali or I know if you have any questions about the program. The Outreachy organizers (Karen Sandler, Sage Sharp, Marina Zhurakhinskaya, Cindy Pallares, and Tony Sebro) can all be reached through our contact form: https://www.outreachy.org/contact/contact-us/. We hope you'll help us spread the word about Outreachy internships! -- Mike Perez

1 0

Ceph 14.2.8 OSD/Pool Nearfull
by Matt Dunavant 22 Feb '21

22 Feb '21

Hi all, We've recently run into an issue where our single ceph rbd pool is throwing errors for nearfull osds. The OSDs themselves vary in PGs/%full with a low of 64/78% and a high of 73/86%. Is there any suggestions on how to get this to balance a little more cleanly? Currently we have 360 drives in a single pool with 8192 PGs. I think we may be able to double the PG num and that will balance things a bit cleaner but I just wanted to see if there may be anything the community suggests other than that. Let me know if there's any further info I forgot to provide if that'll help sort this out. Thanks, RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 741 TiB 135 TiB 606 TiB 607 TiB 81.85 TOTAL 741 TiB 135 TiB 606 TiB 607 TiB 81.85 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL pool 1 162 TiB 46.81M 494 TiB 89.02 20 TiB cluster: health: HEALTH_WARN 85 nearfull osd(s) 1 pool(s) nearfull services: osd: 360 osds: 360 up (since 7d), 360 in (since 7d) data: pools: 1 pools, 8192 pgs objects: 46.81M objects, 169 TiB usage: 607 TiB used, 135 TiB / 741 TiB avail pgs: 8192 active+clean

1 0

Gradually Increasing PG/PGP
by Mark Johnson 22 Feb '21

22 Feb '21

Hi, Probably a basic/stupid question but I'm asking anyway. Through lack of knowledge and experience at the time, when we set up our pools, our pool that holds the majority of our data was created with a PG/PGP num of 64. As the amount of data has grown, this has started causing issues with balance of data across OSDs. I want to increase the PG count to at least 512, or maybe 1024 - obviously, I want to do this incrementally. However, rather than going from 64 to 128, then 256 etc, I'm considering doing this in much smaller increments over a longer period of time so that it will hopefully be doing the majority of moving around of data during the quieter time of day. So, may start by going in increments of 4 until I get up to 128 and then go in jumps of 8 and so on. My question is, will I still end up with the same net result going in increments of 4 until I hit 128 as I would if I were to go straight to 128 in one hit. What I mean by that is that once I reach 128, would I have the exact same level of data balance across PGs as I would if I went straight to 128? Are there any drawbacks in going up in small increments over a long period of time? I know that I'll have uneven PG sizes until I get to that exponent of 2 but that should be OK as long as the end result is the desired result. I suspect I may have a greater amount of data moving around overall doing it this way but given my goal is to reduce the amount of intensive data moves during higher traffic times, that's not a huge concern in the grand scheme of things. Thanks in advance, Mark

2 1

RBD Image can't be formatted - blk_error
by Gaël THEROND 21 Feb '21

21 Feb '21

Hi everyone! I'm facing a weird issue with one of my CEPH clusters: OS: CentOS - 8.2.2004 (Core) CEPH: Nautilus 14.2.11 - stable RBD using erasure code profile (K=3; m=2) When I want to format one of my RBD image (client side) I've got the following kernel messages multiple time with different sector IDs: *[2417011.790154] blk_update_request: I/O error, dev rbd23, sector 164743869184 op 0x3:(DISCARD) flags 0x4000 phys_seg 1 prio class 0[2417011.791404] rbd: rbd23: discard at objno 20110336 2490368~1703936 result -1 * At first I thought about a faulty disk BUT the monitoring system is not showing anything faulty so I decided to run manual tests on all my OSDs to look at disk health using smartctl etc. None of them is marked as not healthy and actually they don't get any counter with faulty sectors/read or writes and the Wear Level is 99% So, the only particularity of this image is it is a 80Tb image, but it shouldn't be an issue as we already have that kind of image size used on another pool. If anyone have a clue at how I could sort this out, I'll be more than happy ^^ Kind regards!

2 5

How to get ceph-volume to take pre-existing, working auth?
by Philip Brown 20 Feb '21

20 Feb '21

I'm trying to use ceph-volume to do various things. It works fine locally, for things like ceph-volume lvm zap But when I want it to do OSD level things, it is unhappy. To use a trivial example, it wants to do things like /usr/bin/ceph --cluster ceph --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json but then dies saying, [errno 13] RADOS permission denied (error connecting to the cluster) and if I directly run that long command myself, it indeed dies. (which is not too surprising, since /var/lib/ceph/bootstrap-osd/ceph.keyring does not exist) However, if I just run from the same command prompt, /usr/bin/ceph osd tree -f json it works fine. How can i get ceph-volume to just use the creds that are already working somewhere? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 1

Re: Data Missing with RBD-Mirror
by Vikas Rana 19 Feb '21

19 Feb '21

Friends, Any help or suggestion here for missing data? Thanks, -Vikas From: Vikas Rana <vrana(a)vtiersys.com> Sent: Tuesday, February 16, 2021 12:20 PM To: 'ceph-users(a)ceph.io' <ceph-users(a)ceph.io> Subject: Data Missing with RBD-Mirror Hi Friends, We have a very weird issue with rbd-mirror replication. As per the command output, we are in sync but the OSD usage on DR side doesn't match the Prod Side. On Prod, we are using close to 52TB but on DR side we are only 22TB. We took a snap on Prod and mounted the snap on DR side and compared the data and we found lot of missing data. Please see the output below. Please help us resolve this issue or point us in right direction. Thanks, -Vikas DR# rbd --cluster cephdr mirror pool status cifs --verbose health: OK images: 1 total 1 replaying research_data: global_id: 69656449-61b8-446e-8b1e-6cf9bd57d94a state: up+replaying description: replaying, master_position=[object_number=390133, tag_tid=4, entry_tid=447832541], mirror_position=[object_number=390133, tag_tid=4, entry_tid=447832541], entries_behind_master=0 last_update: 2021-01-29 15:10:13 DR# ceph osd pool ls detail pool 5 'cifs' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 1294 flags hashpspool stripe_width 0 application rbd removed_snaps [1~5] PROD# ceph df detail POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED cifs 17 N/A N/A 26.0TiB 30.10 60.4TiB 6860550 6.86M 873MiB 509MiB 52.1TiB DR# ceph df detail POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED cifs 5 N/A N/A 11.4TiB 15.78 60.9TiB 3043260 3.04M 2.65MiB 431MiB 22.8TiB PROD#:/vol/research_data# du -sh * 11T Flab1 346G KLab 1.5T More 4.4T ReLabs 4.0T WLab DR#:/vol/research_data# du -sh * 2.6T Flab1 14G KLab 52K More 8.0K RLabs 202M WLab

2 1

ceph orch and mixed SSD/rotating disks
by Philip Brown 18 Feb '21

18 Feb '21

I'm coming back to trying mixed SSD+spinning disks after maybe a year. It was my vague recollection, that if you told ceph "go auto configure all the disks", it would actually automatically carve up the SSDs into the appropriate number of LVM segments, and use them as WAL devices for each hdd based OSD on the system. Was I wrong? Because when I tried to bring up a brand new cluster (Octopus, cephadm bootstrapped), with multiple nodes and multiple disks per node... it seemed to bring up the SSDS as just another set of OSDs. it clearly recognized them as ssd. The output of "ceph orch device ls" showed them as ssd vs hdd for the others. It just...didnt use them as I expected. ? Maybe I was thinking of ceph ansible. Is there not a nice way to do this with the new cephadm based "ceph orch"? I would rather not have to go write json files or whatever by hand, when a computer should be perfectly capable of auto generating this stuff itself -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 2

how to turn off lingering all-available-devices
by Philip Brown 18 Feb '21

18 Feb '21

At one point in the life cycle of my test ceph cluster, I used the --all-available-devices flag of ceph orch. Which will always attempt to bring up any new autodetected disks. I now see in the docs, "If you want to avoid this behavior (disable automatic creation of OSD on available devices), use the unmanaged parameter:" But... I believe I did run it with the --unmanaged flag afterwards. Unfortunately, the original still seems to persist, and it keeps auto creating. How can I get it to stop? I also see mention that, "When the parameter all-available-devices or a DriveGroup specification is used, a cephadm service is created" However, using "ceph orch ps", I dont see any relevantly named service. Where else should I be looking? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 0

rbd move between pools
by Marc 18 Feb '21

18 Feb '21

What is the best way to move an rbd image to a different pool. I want to move some 'old' images (some have snapshots) to backup pool. For some there is also a difference in device class.

3 6

leave
by Hamidreza Hosseini 18 Feb '21

18 Feb '21

1 0

Re: RGW/Swift 404 error when listing/deleting a newly created empty bucket
by Mike Cave 17 Feb '21

17 Feb '21

I am bumping this email to hopefully get some more eyes on it. We are continuing to have this problem. Unfortunately the cluster is very lightly used currently until we go full production so we do not have the level of traffic that would generate a lot of statistics. We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to correlate with when the errors started popping up. Our current plan is to roll back the version to 14.2.10 again and rerun the test that causes the issue. I noted there was another email thread regarding latencies for a user who also updated to 14.2.16 recently and I'm not sure if this could be related or not to my issue. Any suggestions you may have are very welcomed. Cheers, -- Mike Cave On 2021-02-11, 8:37 AM, "Mike Cave" <mcave(a)uvic.ca> wrote: So, as the subject states I have an issue with buckets returning a 404 error when they are listed immediately after being created; as well the bucket fails to be deleted if you try to delete it immediately after creation. The behaviour is intermittent. If I leave the bucket in place for a few minutes, the bucket behaves normally. I’m thinking this is a metadata issue or something along those lines but I’m out of my depth now. To the best of our knowledge the cluster has not changed in any way since the same tests were run in December with no errors. We are running Ceph 14.2.16 on all parts of the cluster. I am using the python-swift client for the connection on a CentOS7 machine. Can replicate the results from the mons or an external client as well. I’m willing to share my test script as well if you would like to see how I’m generating the error. Here is a piece of the logs in case I missed something in the interpretation (log level at 20): 14:23:17.069 7faba00df700 1 ====== starting new request req=0x55fb7a138700 ===== 14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id = tx000000000000000000094-0060245cd5-2b8949-default 14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7 14:23:17.069 7faba00df700 10 host=<NameRemoved> 14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 14:23:17.069 7faba00df700 -1 res_query() failed 14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/404test 14:23:17.069 7faba00df700 10 ver=v1 first=404test req= 14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket scheduling with dmclock client=3 cost=1 14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying requester 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::TempURLEngine denied with reason=-13 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket swift_user=xmcc:swift 14:23:17.069 7faba00df700 20 build_token token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::SignedTokenEngine granted access 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket normalizing buckets and tenants 14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init permissions 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+root+404test : hit (negative entry) 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket recalculating target 14:23:17.069 7faba00df700 10 Starting retarget 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket reading permissions 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op mask 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket required_mask= 4 user.op_mask=7 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op permissions 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- Getting permissions begin with perm_mask=50 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50 14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc 14:23:17.069 7faba00df700 5 Found permission: 15 14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op params 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket pre-executing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket executing 14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: bucket 404test not found 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket completing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op status=-2002 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http status=404 14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op status=-2002 http_status=404 latency=0s ====== -- Mike Cave I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 4

OSDs cannot join, MON leader at 100%
by Frank Schilder 17 Feb '21

17 Feb '21

Dear cephers, I was doing some maintenance yesterday involving shutdown-power up cycles of ceph servers. With the last server I run into a problem. The server runs an MDS and a couple of OSDs. After reboot, the MDS joined the MDS cluster without problems, but the OSDs didn't come up. This was 1 out of 12 servers and I had no such problems with the other 11. I also observed that "ceph status" was responding very slow. Upon further inspection, I found out that 2 of my 3 MONs (the leader and a peon) were running at 100% CPU. Client I/O was continuing, probably because the last cluster map remained valid. On our node performance monitoring I could see that the 2 busy MONs were showing extraordinary network activity. This state lasted for over one hour. After the MONs settled down, the OSDs finally joined as well and everything went back to normal. The other instance I have seen similar behaviour was, when I restarted a MON on an empty disk and the re-sync was extremely slow due to a too large value for mon_sync_max_payload_size. This time, I'm pretty sure it was MON-client communication; see below. Are there any settings similar to mon_sync_max_payload_size that could influence responsiveness of MONs in a similar way? Why do I suspect it is MON-client communication? In our monitoring, I do not see the huge amount of packages sent by the MONs arriving at any other ceph daemon. They seem to be distributed over client nodes, but since we have a large count of client nodes (>550) this is covered by the background network traffic. A second clue is that I have had such extended lock-ups before and, whenever I checked, I only observed these in case the leader had a large share of client sessions. For example, yesterday the client session count per MON was: ceph-01: 1339 (leader) ceph-02: 189 (peon) ceph-03: 839 (peon) I usually restart the leader when such a critical distribution occurs. As long as the leader has the fewest client sessions, I never observe this problem. Ceph version is 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable). Thanks for any clues! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 5

after update to 14.2.16 osd daemons begin to crash
by Boris Behrens 17 Feb '21

17 Feb '21

Hi, currently we experience osd daemon crashes and I can't pin the issue. I hope someone can help me with it. * We operate multiple cluster (440 SSD - 1PB, 36 SSD - 126TB, 40SSD 100TB, 84HDD - 680TB) * All clusters were updated around the same time (2021-02-03) * We restarted ALL ceph daemons (systemctl restart ceph.target) on 2021-02-11 after we added OOMScoreAdjust=-900 the all service files. now in our main cluster (440SSD with 1PB) the OSD daemons begin to crash: # ceph crash ls ID ENTITY NEW 2020-03-06_17:37:54.031675Z_0bbbb807-ff2f-46df-9508-58d319b89bd6 osd.397 2020-05-28_12:23:27.677741Z_061f2449-9a36-4747-a2f8-624e72cd1ad0 osd.410 2021-02-05_07:03:35.943384Z_dffab245-4788-4de2-a677-76b735d5fc01 osd.403 2021-02-15_15:41:27.934194Z_97b57f8f-58f2-4390-9d3e-993874e0e000 osd.395 2021-02-15_18:01:19.774879Z_18160e65-4659-451f-8aae-def2984f1f29 osd.178 2021-02-17_04:51:05.101052Z_9f04c6e8-d0c7-442c-9a38-33d5164d2a83 osd.384 osd.384 and osd.395 are on the same node, which had some memory issues we fixed 2021-02-16_12:00:00 osd.384 was marked as out for >24h when the daemon crashed, and there no more misplaced objects in the cluster. Here is the latest crash dump --- begin dump of recent events --- -9999> 2021-02-17 03:31:31.305 7fcf7e136700 1 do_command 'perf dump' 'result is 30067 bytes -9998> 2021-02-17 03:31:31.626 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9997> 2021-02-17 03:31:32.634 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9996> 2021-02-17 03:31:33.639 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9995> 2021-02-17 03:31:34.647 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9994> 2021-02-17 03:31:35.651 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9993> 2021-02-17 03:31:36.654 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9992> 2021-02-17 03:31:37.657 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9991> 2021-02-17 03:31:38.676 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9990> 2021-02-17 03:31:39.680 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9989> 2021-02-17 03:31:40.684 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9988> 2021-02-17 03:31:41.193 7fcf7e136700 1 do_command 'perf dump' ' -9987> 2021-02-17 03:31:41.193 7fcf7e136700 1 do_command 'perf dump' 'result is 30067 bytes <snip> -31> 2021-02-17 05:50:41.158 7fcf7e136700 1 do_command 'perf dump' ' -30> 2021-02-17 05:50:41.159 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -29> 2021-02-17 05:50:41.804 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851831808 unmapped: 987750400 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -28> 2021-02-17 05:50:42.813 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851831808 unmapped: 987750400 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -27> 2021-02-17 05:50:43.820 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -26> 2021-02-17 05:50:44.825 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -25> 2021-02-17 05:50:45.831 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -24> 2021-02-17 05:50:46.837 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -23> 2021-02-17 05:50:47.840 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -22> 2021-02-17 05:50:48.843 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -21> 2021-02-17 05:50:49.847 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -20> 2021-02-17 05:50:50.853 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -19> 2021-02-17 05:50:51.524 7fcf7e136700 1 do_command 'perf dump' ' -18> 2021-02-17 05:50:51.525 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -17> 2021-02-17 05:50:51.859 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -16> 2021-02-17 05:50:52.862 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -15> 2021-02-17 05:50:53.871 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -14> 2021-02-17 05:50:54.875 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -13> 2021-02-17 05:50:55.886 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -12> 2021-02-17 05:50:56.891 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -11> 2021-02-17 05:50:57.905 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -10> 2021-02-17 05:50:58.911 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9> 2021-02-17 05:50:59.917 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -8> 2021-02-17 05:51:00.929 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -7> 2021-02-17 05:51:01.566 7fcf7e136700 1 do_command 'perf dump' ' -6> 2021-02-17 05:51:01.567 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -5> 2021-02-17 05:51:01.935 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -4> 2021-02-17 05:51:02.943 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -3> 2021-02-17 05:51:03.949 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851102720 unmapped: 988479488 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -2> 2021-02-17 05:51:04.967 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851102720 unmapped: 988479488 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -1> 2021-02-17 05:51:05.091 7fcf743e7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/fastbmap_allocator_impl.h: In function 'uint64_t AllocatorLevel02<T>::claim_free_to_right(uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fcf743e7700 time 2021-02-17 05:51:04.998475 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/fastbmap_allocator_impl.h: 572: FAILED ceph_assert(available >= allocated) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x561c84cc2c7d] 2: (()+0x4d8e45) [0x561c84cc2e45] 3: (HybridAllocator::_add_to_tree(unsigned long, unsigned long)+0x49e) [0x561c853167de] 4: (AvlAllocator::_release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x60) [0x561c85310b20] 5: (HybridAllocator::release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x3a) [0x561c853143ca] 6: (BlueStore::_txc_release_alloc(BlueStore::TransContext*)+0x5f) [0x561c851ee83f] 7: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x1be) [0x561c8522f4ae] 8: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xaa) [0x561c8522fe9a] 9: (BlueStore::_kv_finalize_thread()+0x604) [0x561c85232ed4] 10: (BlueStore::KVFinalizeThread::entry()+0xd) [0x561c852625ed] 11: (()+0x7ea5) [0x7fcf840a2ea5] 12: (clone()+0x6d) [0x7fcf82f6596d] 0> 2021-02-17 05:51:05.145 7fcf743e7700 -1 *** Caught signal (Aborted) ** in thread 7fcf743e7700 thread_name:bstore_kv_final ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (()+0xf630) [0x7fcf840aa630] 2: (gsignal()+0x37) [0x7fcf82e9d387] 3: (abort()+0x148) [0x7fcf82e9ea78] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x561c84cc2ccc] 5: (()+0x4d8e45) [0x561c84cc2e45] 6: (HybridAllocator::_add_to_tree(unsigned long, unsigned long)+0x49e) [0x561c853167de] 7: (AvlAllocator::_release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x60) [0x561c85310b20] 8: (HybridAllocator::release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x3a) [0x561c853143ca] 9: (BlueStore::_txc_release_alloc(BlueStore::TransContext*)+0x5f) [0x561c851ee83f] 10: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x1be) [0x561c8522f4ae] 11: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xaa) [0x561c8522fe9a] 12: (BlueStore::_kv_finalize_thread()+0x604) [0x561c85232ed4] 13: (BlueStore::KVFinalizeThread::entry()+0xd) [0x561c852625ed] 14: (()+0x7ea5) [0x7fcf840a2ea5] 15: (clone()+0x6d) [0x7fcf82f6596d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 3

Ceph @ DevConf.CZ
by Mike Perez 17 Feb '21

17 Feb '21

Hi everyone, Ceph will be present at DevConf.CZ, February 18-20 in a joint booth with the Rook Community! https://www.devconf.cz If you're interested in more information about being present at the booth to provide expertise/content/presentations to our audience, please let me know privately. -- Mike Perez

1 0

best use of NVMe drives
by Magnus HAGDORN 17 Feb '21

17 Feb '21

Hi there, we are in the process of growing our Nautilus ceph cluster. Currently, we have 6 nodes, 3 nodes with 2×5.5TB, 6x11TB disks and 8x186GB SSD and 3 nodes with 6×5.5TB and 6×7.5TB disks. All with dual link 10GE NICs. The SSDs are used for the CephFS metadata pool, the hard drives are used for the CephFS data pool. All OSD journals are kept on the drives themselves. Replication level is 3 for both data and metadata pools. The new servers have 12x12TB disks and 1 1.5TB NVMe drive. We expect to get another 3 similar nodes in the near future. My question is what is the most sensible thing to do with the NVMe drives. I would like to increase the replication level of the metadata pool. So my idea was to split the NVMes into say 4 partitions and add them to the metadata pool. Given the size of the drives and the metadata pool usage (~35GB) that seems overkill. Would it make sense to partition the drives further and stick the OSD journals on the NVMEs? Regards magnus The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 1

SUSE POC - Dead in the water
by Schweiss, Chip 16 Feb '21

16 Feb '21

For the past several months I had been building a sizable Ceph cluster that will be up to 10PB with between 20 and 40 OSD servers this year. A few weeks ago I was informed that SUSE is shutting down SES and will no longer be selling it. We haven't licensed our proof of concept cluster that is currently at 14 OSD nodes, but it looks like SUSE is not going to be the answer here. I'm seeking recommendations for consulting help on this project since SUSE has let me down. I have Ceph installed and operating, however, I've been struggling with getting the pool configured properly for CephFS and getting very poor performance. The OSD servers have TLC NVMe for DB, and Optane NVMe for WAL, so I should be seeing decent performance with the current cluster. I'm not opposed to completely switching OS distributions. Ceph on SUSE was our first SUSE installation. Almost everything else we run is on CentOS, but that may change thanks to IBM cannibalizing CentOS. Please reach out to me if you can recommend someone to sell us consulting hours and/or a support contract. -Chip Schweiss chip.schweiss(a)wustl.edu Washington University School of Medicine

4 6

share haproxy config for radosgw
by Marc 16 Feb '21

16 Feb '21

I was wondering if someone could post a config for haproxy. Is there something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.

8 10

Consequences of setting bluestore_fsck_quick_fix_on_mount to false?
by Matthew Vernon 16 Feb '21

16 Feb '21

Hi, Looking at the Octopus upgrade instructions, I see "the first time each OSD starts, it will do a format conversion to improve the accounting for “omap” data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data)." and that I can disable this by setting bluestore_fsck_quick_fix_on_mount to false. A couple of questions about this: i) what are the consequences of turning off this "quick fix"? Is it possible to have it run in the background or similar? ii) is there any way to narrow down the time estimate? Our production cluster has 3060 OSDs on hdd (with block.db on NVME), and obviously 3000 lots of "a few hours" is an awful lot of time... I'll be doing some testing on our test cluster (by putting 10M objects into an S3 bucket before trying the upgrade), but it'd be useful to have some idea of how this is likely to work at scale... Thanks, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

2 2

Upgrading Ceph luminous to mimic on debian-buster
by Jean-Marc FONTANA 16 Feb '21

16 Feb '21

Hello everyone, We just installed a Ceph cluster version luminous (12.2.11) on servers working with Debian buster (10.8) using ceph-deploy and we are trying to upgrade it to mimic but can't find a way to do it. We tried ceph-deploy install --release mimic mon1 mon2 mon3 (after having modified /etc/apt/sources.list.d/ceph.list) but this does nothing because the packets are said to be up to date. Could someone help us, please ? Best regards

2 1

osds processes shutdown during outage
by Marcel Kuiper 16 Feb '21

16 Feb '21

Hi, (sorry if this gets posted twice. I forgot a subject in the first mail) We expereinced an outage this morning on a jewel cluster with 1559 osds. It appeared that a switch uplink in a rack misbehaved and once shutting that interface ceph health restored quickly. I have some questions though on osd behaviour that I hope someone can answer 1 - In a lot of osd logs I saw that neighbours reported the osd down (while the process was still running and obviously logging). Then after a while the logs shows * Got signal Interrupt * prepare_to_stop starting shutdown and the osd process stops Why does the osd proces stop? Is it instructed to do so by the monitor because neighbours reported it down and ceph wants to avoid flapping? 2 - The osds reported a lot of * heartbeat_check: no reply from #ip:#port When I telnet to the ip and port I get a connection just fine. Is there a way to run a heartbeat_check from the commandline so that we can try capture the traffic to determine why it fails Thanks Marcel

1 0

struggling to achieve high bandwidth on Ceph dev cluster - HELP
by Bobby 16 Feb '21

16 Feb '21

Hi, Hello I am using rados bench tool. Currently I am using this tool on the development cluster after running vstart.sh script. It is working fine and I am interested in benchmarking the cluster. However I am struggling to achieve a good bandwidth i.e. bandwidth (MB/sec). My target throughput is at least 50 MB/sec and more. But mostly I am achieving is around 15-20 MB/sec. So, very poor. I am quite sure I am missing something. Either I have to change my cluster through vstart.sh script or I am not fully utilizing the rados bench tool. Or may be both. i.e. not the right cluster and also not using the rados bench tool correctly. Some of the shell examples I have been using to build the cluster are bellow: MDS=0 RGW=1 ../src/vstart.sh -d -l -n --bluestore MDS=0 RGW=1 MON=1 OSD=4../src/vstart.sh -d -l -n --bluestore While using rados bench tool I have been trying with different block sizes 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K. And I have also been changing the -t parameter in the shell to increase concurrent IOs. Looking forward to help. Bobby

3 9

POC Hardware questions
by Oliver Weinmann 16 Feb '21

16 Feb '21

Dear All, A questions that probalby has been asked by many other users before. I want to do a POC. For the POC I can use old decomissioned hardware. Currently I have 3 x IBM X3550 M5 with: 1 Dualport 10G NIC Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz 64GB RAM the other two have a slower CPU but more RAM: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz 512GB RAM Of course I can re-arrange the RAM. The switches are not LACP capable, so I'm planning to use bonding in active-active. For the disks I'm planning on buying 12 x Samsung PM883 1.9TB and use them in an EC pool. My questions are: 1. Which bonding mode should I choose? balance-alb? 2. Are the disks ok for a POC? Or should I rather go with more smaller disks (960GB) e.g. 24 in total? 3. Are there any drawbacks when using EC pools? Workload will be mostly VMs (Vsphere / Openstack), but also CephFS with Samba Gateway. Many thanks

2 1

Re: can't remove osd service by "ceph orch rm <service name>"
by Juan Miguel Olmo Martinez 16 Feb '21

16 Feb '21

Hi Tony. Take a look to: https://docs.ceph.com/en/latest/mgr/orchestrator/#remove-an-osd -- Juan Miguel Olmo Martínez Senior Software Engineer Red Hat <https://www.redhat.com/> jolmomar(a)redhat.com <https://www.redhat.com/>

1 0

can't remove osd service by "ceph orch rm <service name>"
by Tony Liu 16 Feb '21

16 Feb '21

Hi, This is with v15.2 and v15.2.8. Once an OSD service is applied, it can't be removed. It always shows up from "ceph orch ls". "ceph orch rm <osd service name>" only marks it "unmanaged", but not actually removes it. Is this the expected? Thanks! Tony

1 0

Re: Backups of monitor
by Freddy Andersen 15 Feb '21

15 Feb '21

I would say everyone recommends at least 3 monitors and since they need to be 1,3,5 or 7 I always read that as 5 is the best number (if you have 5 servers in your cluster). The other reason is high availability since the MONs use Paxos for the quorum and I like to have 3 in the quorum you need 5 to be able to do maintenance. (2 out of 3, 3 out of 5… ) So if you are doing maintenance on a mon host in a 5 mon cluster you will still have 3 in the quorum. From: huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> Date: Friday, February 12, 2021 at 8:42 AM To: Freddy Andersen <freddy(a)cfandersen.com>, Marc <Marc(a)f1-outsourcing.eu>, Michal Strnad <michal.strnad(a)cesnet.cz>, ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Backups of monitor Why 5 instead of 3 MONs are required? huxiaoyu(a)horebdata.cn From: Freddy Andersen Date: 2021-02-12 16:05 To: huxiaoyu(a)horebdata.cn; Marc; Michal Strnad; ceph-users Subject: Re: [ceph-users] Re: Backups of monitor I would say production should have 5 MON servers From: huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> Date: Friday, February 12, 2021 at 7:59 AM To: Marc <Marc(a)f1-outsourcing.eu>, Michal Strnad <michal.strnad(a)cesnet.cz>, ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Backups of monitor Normally any production Ceph cluster will have at least 3 MONs, does it reall need a backup of MON? samuel huxiaoyu(a)horebdata.cn From: Marc Date: 2021-02-12 14:36 To: Michal Strnad; ceph-users(a)ceph.io Subject: [ceph-users] Re: Backups of monitor So why not create an extra start it only when you want to make a backup, wait until it is up to date, stop it and then stop it to back it up? > -----Original Message----- > From: Michal Strnad <michal.strnad(a)cesnet.cz> > Sent: 11 February 2021 21:15 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Backups of monitor > > Hi all, > > We are looking for a proper solution for backups of monitor (all maps > that they hold). On the internet we found advice that we have to stop > one of monitor, back it up (dump) and start daemon again. But this is > not right approach due to risk of loosing quorum and need of > synchronization after monitor is back online. > > Our goal is to have at least some (recent) metadata of objects in > cluster for the last resort when all monitors are in very bad > shape/state and we could start any of them. Maybe there is another > approach but we are not aware of it. > > We are running the latest nautilus and three monitors on every cluster. > > Ad. We don't want to use more monitors than thee. > > > Thank you > Cheers > Michal > -- > Michal Strnad > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

5 4

CephFS Octopus snapshots / kworker at 100% / kernel vs. fuse client
by Sebastian Knust 15 Feb '21

15 Feb '21

Hi, I am running a Ceph Octopus (15.2.8) cluster primarily for CephFS. Metadata is stored on SSD, data is stored in three different pools on HDD. Currently, I use 22 subvolumes. I am rotating snapshots on 16 subvolumes, all in the same pool, which is the primary data pool for CephFS. Currently I have 41 snapshots per subvolume. The goal is 50 snapshots (see bottom of mail for details). Snapshots are only placed in the root subvolume directory, i.e. /volumes/_nogroup/subvolname/hex-id/.snap I place the snapshots on one of the nodes. Complete CephFS is mounted, mkdir and rmdir is performed for each relevant subvolume, then CephFS is unmounted again. All PGs are active+clean most of the time, only a few in snaptrim for 1-2 minutes after snapshot deletion. I therefore assume that snaptrim is not a limiting factor. Obviously, the total number of snapshots is more than the 400 and 100 I see mentioned in some documentation. I am unsure if that is an issue here, as the snapshots are all in disjunct subvolumes. When mounting the subvolumes with kernel client (ranging from CentOS 7 supplied 3.10 up to 5.4.93), after some time and for some subvolumes the kworker process begins to hug 100% cpu usage and stat operations become very slow (even slower than with fuse client). I can mostly replicate this by starting specific rsync operations (with many small files, e.g. CTAN, CentOS, Debian mirrors) and by running a bareos backup. The kworker process seems to be stuck even after terminating the causing operating, i.e. rsync or bareos-fd. Interestingly, I can even trigger these issues on a host that has only a single CephFS subvolume without any snapshots mounted, as long as that subvolume is in the same pool as other subvolumes with snapshots. I don't see any abnormal behaviour on the cluster nodes or on other clients during these kworker hanging phases. With fuse client, in normal operation stat calls are about 10-20x slower than with the kernel client. However, I don't encounter the extreme slowdown behaviour. I am therefore currently mounting some known-problematic subvolumes with fuse and non-problematic subvolumes with the kernel client. My questions are: - Is this known or expected behaviour? - I could move the subvolumes with snapshots into a subvolumegroup and snapshot the whole group instead of each subvolume. Will this be likely to solve the issues? - What is the current recommendation regarding CephFS and max number of snapshots? Cluster setup: 5 nodes with a total of 56 OSDs Each node has a Xeon Silver 4208 and 128 GB RAM Each node has two 480GB Samsung PM883 SSD used for CephFS metadata pool HDDs are ranging from 8TB to 14TB, majority is 14TB 10 GbE internal network and 10 GbE client network, no Jumbo frames $ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 520 TiB 141 TiB 378 TiB 379 TiB 72.88 ssd 3.9 TiB 3.8 TiB 1.7 GiB 97 GiB 2.46 TOTAL 524 TiB 145 TiB 378 TiB 379 TiB 72.36 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 66 MiB 57 198 MiB 0 23 TiB cephfs.cephfs.meta 2 1024 26 GiB 2.29M 77 GiB 2.06 1.2 TiB cephfs.cephfs.data 3 1024 70 TiB 54.95M 213 TiB 75.19 23 TiB lofar 4 512 77 TiB 21.41M 154 TiB 68.68 35 TiB proxmox 6 64 526 GiB 158.60k 1.6 TiB 2.16 23 TiB archive 7 32 7.3 TiB 5.42M 10 TiB 12.57 56 TiB Snapshots are only on cephfs.cephfs.data pool. Intended snapshot rotation: 4 quarter-hourly snapshots 24 hourly snapshots 14 daily snapshots 8 weekly snapshots Cheers Sebastian

2 1

Is replacing OSD whose data is on HDD and DB is on SSD supported?
by Tony Liu 15 Feb '21

15 Feb '21

Hi, I've been trying with v15.2 and v15.2.8, no luck. Wondering if this is actually supported or ever worked for anyone? Here is what I've done. 1) Create a cluster with 1 controller (mon and mgr) and 3 OSD nodes, each of which is with 1 SSD for DB and 8 HDDs for data. 2) OSD service spec. service_type: osd service_id: osd-spec placement: hosts: - ceph-osd-1 - ceph-osd-2 - ceph-osd-3 spec: block_db_size: 92341796864 data_devices: model: ST16000NM010G db_devices: model: KPM5XRUG960G 3) Add OSD hosts and apply OSD service spec. 8 OSDs (data on HDD and DB on SSD) are created on each host properly. 4) Run "orch osd rm 1 --replace --force". OSD is marked "destroyed" and reweight is set to 0 in "osd tree". "pg dump" shows no PG on that OSD. "orch ps" shows no daemon running for that OSD. 5) Run "orch device zap <host> <device>". VG and LV for HDD are removed. LV for DB stays. "orch device ls" shows HDD device is available. 6) Cephadm finds OSD claims and applies OSD spec on the host. Here is the message. ============================ cephadm [INF] Found osd claims -> {'ceph-osd-1': ['1']} cephadm [INF] Found osd claims for drivegroup osd-spec -> {'ceph-osd-1': ['1']} cephadm [INF] Applying osd-spec on host ceph-osd-1... cephadm [INF] Applying osd-spec on host ceph-osd-2... cephadm [INF] Applying osd-spec on host ceph-osd-3... cephadm [INF] ceph-osd-1: lvm batch --no-auto /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj --db-devices /dev/sdb --block-db-size 92341796864 --osd-ids 1 --yes --no-systemd code: 0 out: [''] err: ['/bin/docker:stderr --> passed data devices: 8 physical, 0 LVM', '/bin/docker:stderr --> relative data size: 1.0', '/bin/docker:stderr --> passed block_db devices: 1 physical, 0 LVM', '/bin/docker:stderr --> 1 fast devices were passed, but none are available'] ============================ Q1. Is DB LV on SSD supposed to be deleted or not, when replacing an OSD whose data is on HDD and DB is on SSD? Q2. If yes from Q1, is a new DB LV supposed to be created on SSD as long as there is sufficient free space, when building the new OSD? Q3. If no from Q1, since it's replacing, is the old DB LV going to be reused for the new OSD? Again, is this actually supposed to work? Am I missing anything or just trying on some unsupported feature? Thanks! Tony

1 1

reinstalling node with orchestrator/cephadm
by Kenneth Waegeman 15 Feb '21

15 Feb '21

Hi all, I'm running a 15.2.8 cluster using ceph orch with all daemons adopted to cephadm. I tried reinstall an OSD node. Is there a way to make ceph orch/cephadm activate the devices on this node again, ideally automatically? I tried running `cephadm ceph-volume -- lvm activate --all` but this has an error related to dmcrypt: > [root@osd2803 ~]# cephadm ceph-volume -- lvm activate --all > Using recent ceph image docker.io/ceph/ceph:v15 > /usr/bin/podman:stderr --> Activating OSD ID 0 FSID > 697698fd-3fa0-480f-807b-68492bd292bf > /usr/bin/podman:stderr Running command: /usr/bin/mount -t tmpfs tmpfs > /var/lib/ceph/osd/ceph-0 > /usr/bin/podman:stderr Running command: /usr/bin/ceph-authtool > /var/lib/ceph/osd/ceph-0/lockbox.keyring --create-keyring --name > client.osd-lockbox.697698fd-3fa0-480f-807b-68492bd292bf --add-key > AQAy7Bdg0jQsBhAAj0gcteTEbcpwNNvMGZqTTg== > /usr/bin/podman:stderr stdout: creating > /var/lib/ceph/osd/ceph-0/lockbox.keyring > /usr/bin/podman:stderr added entity > client.osd-lockbox.697698fd-3fa0-480f-807b-68492bd292bf > auth(key=AQAy7Bdg0jQsBhAAj0gcteTEbcpwNNvMGZqTTg==) > /usr/bin/podman:stderr Running command: /usr/bin/chown -R ceph:ceph > /var/lib/ceph/osd/ceph-0/lockbox.keyring > /usr/bin/podman:stderr Running command: /usr/bin/ceph --cluster ceph > --name client.osd-lockbox.697698fd-3fa0-480f-807b-68492bd292bf > --keyring /var/lib/ceph/osd/ceph-0/lockbox.keyring config-key get > dm-crypt/osd/697698fd-3fa0-480f-807b-68492bd292bf/luks > /usr/bin/podman:stderr stderr: Error initializing cluster client: > ObjectNotFound('RADOS object not found (error calling conf_read_file)',) > /usr/bin/podman:stderr --> RuntimeError: Unable to retrieve dmcrypt > secret > Traceback (most recent call last): > File "/usr/sbin/cephadm", line 6111, in <module> > r = args.func() > File "/usr/sbin/cephadm", line 1322, in _infer_fsid > return func() > File "/usr/sbin/cephadm", line 1381, in _infer_image > return func() > File "/usr/sbin/cephadm", line 3611, in command_ceph_volume > out, err, code = call_throws(c.run_cmd(), verbose=True) > File "/usr/sbin/cephadm", line 1060, in call_throws > raise RuntimeError('Failed command: %s' % ' '.join(command)) > RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host > --net=host --entrypoint /usr/sbin/ceph-volume --privileged > --group-add=disk -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e > NODE_NAME=osd2803.banette.os -v /dev:/dev -v /run/udev:/run/udev -v > /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm > docker.io/ceph/ceph:v15 lvm activate --all The OSDs are encrypted indeed. `cephadm ceph-volume lvm list` and `cephadm shell ceph -s` run just fine, and if I run ceph-volume directly, the same command works, but then of course the daemons are started in the legacy way again, not in containers. Is there another way trough the 'ceph orch' to achieve this? Or if `cephadm ceph-volume -- lvm activate --all` would be the way to go here, I'm probably seeing a bug here ? Thanks!! Kenneth

4 7

Multisite cluster sync latency
by Szabo, Istvan (Agoda) 15 Feb '21

15 Feb '21

Hi Gents, Can you tell me how much is your latency in your multisite cluster? Multisite should be latency sensitive and I scare this is my sync issue, but I don't really know what means "low latency". Here is mine, I wonder is it good or not. In HKG: "data-sync-from-ash": { "fetch_bytes": { "avgcount": 0, "sum": 0 }, "fetch_not_modified": 425395, "fetch_errors": 1, "poll_latency": { "avgcount": 890, "sum": 47481.671537512, "avgtime": 53.350192738 }, "poll_errors": 0 }, "data-sync-from-sin": { "fetch_bytes": { "avgcount": 0, "sum": 0 }, "fetch_not_modified": 484757, "fetch_errors": 0, "poll_latency": { "avgcount": 21686, "sum": 135649.750753768, "avgtime": 6.255176185 }, "poll_errors": 3 In ASH: "data-sync-from-hkg": { "fetch_bytes": { "avgcount": 7904, "sum": 497898243 }, "fetch_not_modified": 7383973, "fetch_errors": 654, "poll_latency": { "avgcount": 6586, "sum": 2568055.690045521, "avgtime": 389.926463717 }, "poll_errors": 3 }, "data-sync-from-sin": { "fetch_bytes": { "avgcount": 13362, "sum": 800114616 }, "fetch_not_modified": 7326406, "fetch_errors": 558, "poll_latency": { "avgcount": 10137, "sum": 3145053.032619919, "avgtime": 310.254812333 }, "poll_errors": 5 }, In SGP: "data-sync-from-ash": { "fetch_bytes": { "avgcount": 0, "sum": 0 }, "fetch_not_modified": 2057839, "fetch_errors": 1, "poll_latency": { "avgcount": 8874, "sum": 682176.718044618, "avgtime": 76.873644133 }, "poll_errors": 0 }, "data-sync-from-hkg": { "fetch_bytes": { "avgcount": 114, "sum": 1097512 }, "fetch_not_modified": 1939198, "fetch_errors": 823, "poll_latency": { "avgcount": 2123, "sum": 60947.760976996, "avgtime": 28.708318877 }, "poll_errors": 1 ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Latency increase after upgrade 14.2.8 to 14.2.16
by Björn Dolkemeier 15 Feb '21

15 Feb '21

Hi, after upgrading Ceph from 14.2.8 to 14.2.16 we experienced increased latencies. There were no changes in hardware, configuration, workload or networking, just a rolling-update via ceph-ansible on running production cluster. The cluster consists of 16 OSDs (all SSD) over 4 Nodes. The VMs served via RBD from this cluster currently suffer on i/o wait cpu. These are some latencies that are increased after the update: - op_r_latency - op_w_latency - kv_final_lat - state_kv_commiting_lat - submit_lat - subop_w_latency Do these latencies point to KV/RocksDB? These are some latencies which are NOT increased after the update: - kv_sync_lat - kv_flush_lat - kv_commit_lat I attached one graph showing the massive increase after the update. I tried setting bluefs_buffered_io=true (as it’s default value was changed and it was mentioned as performance relevant) for all OSDs in one host but this does not make a difference. The ceph.conf is fairly simple: [global] cluster network = xxx fsid = xxx mon host = xxx public network = xxx [osd] osd memory target = 10141014425 Any ideas what to try? Help appreciated. Björn -- dbap GmbH phone +49 251 609979-0 / fax +49 251 609979-99 Heinr.-von-Kleist-Str. 47, 48161 Muenster, Germany http://www.dbap.de dbap GmbH, Sitz: Muenster HRB 5891, Amtsgericht Muenster Geschaeftsfuehrer: Bjoern Dolkemeier

2 6

adding a second rgw instance on the same zone
by Adrian Nicolae 13 Feb '21

13 Feb '21

Hi guys, I have a Mimic cluster with only one RGW machine. My setup is simple - one realm, one zonegroup, one zone. How can I safely add a second RGW server to the same zone ? Is it safe to just run "ceph-deploy rgw create" for the second server without impacting the existing metadata pools ? What about the existing S3/Swift users - they should be available to the second RGW from the current pools, right ? My biggest concern is that the second RGW server will try to recreate some internal pools when going online so I just want to double-check that I will not mess the current setup when adding the second instance :) Thanks.

2 1

Increasing QD=1 performance (lowering latency)
by Wido den Hollander 12 Feb '21

12 Feb '21

Hi, There are many talks and presentations out there about Ceph's performance. Ceph is great when it comes to parallel I/O, large queue depths and many applications sending I/O towards Ceph. One thing where Ceph isn't the fastest are 4k blocks written at Queue Depth 1. Some applications benefit very much from high performance/low latency I/O at qd=1, for example Single Threaded applications which are writing small files inside a VM running on RBD. With some tuning you can get to a ~700us latency for a 4k write with qd=1 (Replication, size=3) I benchmark this using fio: $ fio --ioengine=librbd --bs=4k --iodepth=1 --direct=1 .. .. .. .. 700us latency means the result will be about ~1500 IOps (1000 / 0.7) When comparing this to let's say a BSD machine running ZFS that's on the low side. With ZFS+NVMe you'll be able to reach about somewhere between 7.000 and 10.000 IOps, the latency is simply much lower. My benchmarking / test setup for this: - Ceph Nautilus/Octopus (doesn't make a big difference) - 3x SuperMicro 1U with: - AMD Epyc 7302P 16-core CPU - 128GB DDR4 - 10x Samsung PM983 3,84TB - 10Gbit Base-T networking Things to configure/tune: - C-State pinning to 1 - CPU governer to performance - Turn off all logging in Ceph (debug_osd, debug_ms, debug_bluestore=0) Higher clock speeds (New AMD Epyc coming in March!) help to reduce the latency and going towards 25Gbit/100Gbit might help as well. These are however only very small increments and might help to reduce the latency by another 15% or so. It doesn't bring us anywhere near the 10k IOps other applications can do. And I totally understand that replication over a TCP/IP network takes time and thus increases latency. The Crimson project [0] is aiming to lower the latency with many things like DPDK and SPDK, but this is far from finished and production ready. In the meantime, am I overseeing some things here? Can we reduce the latency further of the current OSDs? Reaching a ~500us latency would already be great! Thanks, Wido [0]: https://docs.ceph.com/en/latest/dev/crimson/crimson/

5 4

Storage-class split objects
by Marcelo 12 Feb '21

12 Feb '21

Hello all! We have a cluster where there are HDDs for data and NVMEs for journals and indexes. We recently added pure SSD hosts, and created a storage class SSD. To do this, we create a default.rgw.hot.data pool, associate a crush rule using SSD and create a HOT storage class in the placement-target. The problem is when we send an object to use a HOT storage class, it is in both the STANDARD storage class pool and the HOT pool. STANDARD pool: # rados -p default.rgw.buckets.data ls d86dade5-d401-427b-870a-0670ec3ecb65.385198.4_LICENSE # rados -p default.rgw.buckets.data stat d86dade5-d401-427b-870a-0670ec3ecb65.385198.4_LICENSE default.rgw.buckets.data/d86dade5-d401-427b-870a-0670ec3ecb65.385198.4_LICENSE mtime 2021-02-09 14: 54: 14.000000, size 0 HOT pool: # rados -p default.rgw.hot.data ls d86dade5-d401-427b-870a-0670ec3ecb65.385198.4__shadow_.rmpla1NTgArcUQdSLpW4qEgTDlbhn9f_0 # rados -p default.rgw.hot.data stat d86dade5-d401-427b-870a-0670ec3ecb65.385198.4__shadow_.rmpla1NTgArcUQdSLpW4qEgTDlbhn9f_0 default.rgw.hot.data/d86dade5-d401-427b-870a-0670ec3ecb65.385198.4__shadow_.rmpla1NTgArcUQdSLpW4qEgTDlbhn9f_0 mtime 2021-02-09 14: 54: 14.000000, size 15220 The object itself is in the HOT pool, however it creates this other object similar to an index in the STANDARD pool. Monitoring with iostat we noticed that this behavior generates an unnecessary IO on disks that do not need to be touched. Why this behavior? Are there any ways around it? Thanks, Marcelo

2 4

Re: Backups of monitor
by Freddy Andersen 12 Feb '21

12 Feb '21

I would say production should have 5 MON servers From: huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> Date: Friday, February 12, 2021 at 7:59 AM To: Marc <Marc(a)f1-outsourcing.eu>, Michal Strnad <michal.strnad(a)cesnet.cz>, ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Backups of monitor Normally any production Ceph cluster will have at least 3 MONs, does it reall need a backup of MON? samuel huxiaoyu(a)horebdata.cn From: Marc Date: 2021-02-12 14:36 To: Michal Strnad; ceph-users(a)ceph.io Subject: [ceph-users] Re: Backups of monitor So why not create an extra start it only when you want to make a backup, wait until it is up to date, stop it and then stop it to back it up? > -----Original Message----- > From: Michal Strnad <michal.strnad(a)cesnet.cz> > Sent: 11 February 2021 21:15 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Backups of monitor > > Hi all, > > We are looking for a proper solution for backups of monitor (all maps > that they hold). On the internet we found advice that we have to stop > one of monitor, back it up (dump) and start daemon again. But this is > not right approach due to risk of loosing quorum and need of > synchronization after monitor is back online. > > Our goal is to have at least some (recent) metadata of objects in > cluster for the last resort when all monitors are in very bad > shape/state and we could start any of them. Maybe there is another > approach but we are not aware of it. > > We are running the latest nautilus and three monitors on every cluster. > > Ad. We don't want to use more monitors than thee. > > > Thank you > Cheers > Michal > -- > Michal Strnad > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Backups of monitor
by Michal Strnad 12 Feb '21

12 Feb '21

Hi all, We are looking for a proper solution for backups of monitor (all maps that they hold). On the internet we found advice that we have to stop one of monitor, back it up (dump) and start daemon again. But this is not right approach due to risk of loosing quorum and need of synchronization after monitor is back online. Our goal is to have at least some (recent) metadata of objects in cluster for the last resort when all monitors are in very bad shape/state and we could start any of them. Maybe there is another approach but we are not aware of it. We are running the latest nautilus and three monitors on every cluster. Ad. We don't want to use more monitors than thee. Thank you Cheers Michal -- Michal Strnad

3 2

Re: Cannot access "Object Gateway" in dashboard after setting rgw api keys
by Alfonso Martinez Hidalgo 12 Feb '21

12 Feb '21

Hi Troels, 1) It seems you need to set up the user id like this: ceph dashboard set-rgw-api-user-id <user_id> More info here: https://docs.ceph.com/en/nautilus/mgr/dashboard/#enabling-the-object-gatewa… 2) Have you set up multisite configuration (realms/zonegroups/zones) ? Please paste the output of: radosgw-admin realm list radosgw-admin zonegroup list radosgw-admin zone list Regards, -- Alfonso Martínez Senior Software Engineer, Ceph Storage Red Hat <https://www.redhat.com> <https://red.ht/sig>

2 1

Network design issues
by Frank Schilder 12 Feb '21

12 Feb '21

Dear cephers, I believe we are facing a bottleneck due to an inappropriate overall network design and would like to hear about experience and recommendations. I start with a description of the urgent problem/question and follow up with more details/questions. These observations are on our HPC home file system served with ceph. It has 12 storage servers facing 550+ client servers. Under high load, I start seeing "slow ping time" warnings with quite incredible latencies. I suspect we have a network bottleneck. On the storage servers we have 6x10G LACP trunks. Clients are on single 10G NICs. We have separate VLANs for front- and back network, but they both go through all NICs in the same way, so, technically, its just one cluster network shared with clients. The aggregated bandwidth is sufficient for a single-node storage server load (roughly matches with disk controller IO capacity). However, point-to-point connections are 10G only and I believe that we start observing clients saturating a 10G link and starving all other ceph cluster traffic that needs to go through this link as well. This, in turn, leads to backlog effects with slow ops on unrelated OSDs, affecting overall user experience. The number of OSDs reporting slow ping times is about the percentage one would expect if one or two 10G links are congested. Its usually just one storage server that coughs up. I guess the users with aggressive workloads getting the full bandwidth are happy, but everyone else is complaining. What I observe is that one or two clients can DOS everyone else. I typically see a very high read bandwidth from a few OSDs only and my suspicion is that this is a large job of 50-100 nodes starting the same application at the same time. For example, 50-100 clients reading the same executable simultaneously. I see 5-6GB/s and up to 10K IOP/s read, which is really good in principle. Except that is not fair-shared with other users. Question: I start considering to enable QOS on the switches for traffic between storage servers and would like to know if anyone is doing this and what the experience is. Unfortunately, our network design is probably flawed and makes this now difficult; see below. More Info. Our FS data pool is EC 8+2. I have fast-read enabled. Hence, the network traffic amplification for both, read and write, is quite substantial. Our network is a spine-leaf architecture where ceph servers and ceph clients are distributed more or less equally over the leaf switches. I'm afraid that this is a first flaw in the design, because storage servers and clients compete for the same switches and the clients greatly outnumber the storage servers. It also makes implementing QOS a real pain while it could be just traffic shaping on an uplink trunk to clients if the storage servers were isolated. This is the first design question: Isolated storage cluster providing service via uplinks/gateways versus "integrated/hyper-converged" where storage servers and clients are distributed equally over a spine-leaf architecture. Pros and cons? We have a 100G spine VLT-pair with ports configured as 40G. Up-links from leafs are 2x40, in fact, we have these leafs configured as VLT-pairs for HA as well. A pair has 2x2x40G uplinks and 2x40G VLT interlinks. There are 2 ceph servers per VLT leaf-pair and ca. 85+ client servers on the same pair. There are also clients on leaf switches without ceph servers. I don't think the 40G uplinks are congested, but you never know. We started with the ceph servers having 15HDDs for fs data and 1 SSD for fs meta-data each. With this configuration, the disk speed was the bottleneck and I observed slow ops under high load, but everything was more or less stable. I recently changed an MDS setting that greatly improved both, client performance and also the client's ability to overload OSDs. In addition, one week ago I added 20HDDs in a JBOD per host, which more than doubled the HDD throughput. Both increases in performance together have now the counter-intuitive effect that aggregated performance has tripled in comparison to 2 months ago, but the user experience is very erratic. My suspicion is, as explained above, that each server can now handle a volume of traffic that easily saturates a 10G link, leading to observations that seem to indicate insufficient network capacity whenever too many client/cluster requests go through the same 10G link. In essence, we increased aggregated performance greatly but users complain more than ever. I suspect that this imbalance of server throughput ability and 10G point-to-point limitation is a problem. However, I cannot change the networking and would like some advice of how similar set-ups are configured and if QOS can help. My idea is to enable dot1p layer 2 QOS and give traffic coming from ports with storage servers connected a higher priority than traffic coming from everywhere else. I know it would be a lot simpler if the storage cluster was isolated, but I have to deal with the situation as is for now. Any advice and experience is highly appreciated. If I do it, should I do QOS on both, front- and back network, or is QOS on the VLAN for back-network enough? Note that MONs are only on the front network. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

CEPHFS - MDS gracefull handover of rank 0
by Martin Hronek 12 Feb '21

12 Feb '21

Hello fellow CEPH-users, currently we are updating our CEPH(14.2.16) and making changes to some config settings. TLDR: is there a way to make a graceful MDS active node shutdown without loosing the caps, open files and client connections? Something like handover active state, promote standby to active, ...? Sadly we run into some difficulties when restarting MDS Nodes. While we had two active nodes and one standby we initially though that this would have a nice handover when restarting the active rank ... sadly we saw how the node was going through the states: replay-reconnect-rejoin-active as nicely visualized here https://docs.ceph.com/en/latest/cephfs/mds-states/ This left some nodes going into timeouts until the standby node has gone into the active state again, most probably since the cephfs hast already some 600k folders and 3M files and from the client side it took more than 30s. So before the next MDS the FS config where changed to one active and one standby-replay node, the idea was that since the MDS replay nodes follows the active one the handover would be smoother. The active state was reached faster, but we still noticed some hiccups on the clients while the new active MDS was waiting for clients to reconnect(state up:reconnect) after the failover. The next idea was to do a manual node promotion, graceful shutdown or something similar - where the open caps and sessions would be handed over ... but I did not find any hint in the docs regarding this functionality. But, this should somehow be possible (imho), since when adding a second active mds node (max_mds 2) and then removing it again (max_mds 1) the rank 1 node goes to stopping-state and hands over all clients/caps to rank 0 without interruptions for the clients. Therefore my question: how can one gracefully shutdown an active rank 0 mds node or promote an standby node to the active state without loosing open files/caps or client sessions? Thanks in advance, M

4 6

cephadm upgrade issue persisting with one node
by Darrin Hodges 11 Feb '21

11 Feb '21

Hi all, Still getting upgrade issue with cephadm " Upgrade: failed to pull target image". On each of the nodes in the cluster I can do: docker pull docker.io/ceph/ceph:v15.2.8 And there is no error but the upgrade command fails still. I can see an entry in the logs for: Feb 11 22:27:56 ceph-admin01 bash[2641]: audit 2021-02-11T22:27:55.339997+0000 mon.ceph-admin01 (mon.0) 61 : audit [INF] from='mgr.12064184 ' entity='mgr.ceph-osd01.rpgexq' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"docker.io/ceph/ceph:v15.2.8\", \"progress_id\": \"7cf2e315-6cfe-4e9a-88bc-ec8d611b6b4f\", \"error\": \"UPGRADE_FAILED_PULL: Upgrade: failed to pull target image\", \"paused\": true}"}]: dispatch any ideas how I can find extra info on what is going on there? thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

1 1

ceph osd df results
by Marc 11 Feb '21

11 Feb '21

Should the ceph osd df results not have this result for every device class? I do not think that there people mixing these classes in pools. MIN/MAX VAR: 0.78/4.28 STDDEV: 6.15

2 1

MultiSite Sync Problem and Shard Number Relation?
by George Yil 11 Feb '21

11 Feb '21

Hi, I am sort of newby to RGW multisite. I guess there is an important limitation about bucket index sharding if you run multisite. I would like to learn better or correct myself. And I also want to leave a bookmark here for future cephers if possible. I apologize if this is asked before however I am not able to find a good explanation from the mailing list. AFAIK RGW needs bucket indexes as to serve bucket listing. This is optional but mostly demanded. Now you have to set a value for the number of shards of the bucket. This is not dynamic since multisite does not support dynamic resharding yet. AFAIK if your bucket hits the shard limit then your bucket sync is in trouble or maybe stopped. And you should plan a cluster outage in order to reshard the bucket and resync from start. I would like to verify if this is a correct assumption? Here is the problem I've encounter right now. I have 2 nautilus 14.2.9 RGW multisite clusters. Since I was not fully aware of multisite/bucket limitations, I find myself with a fairly large bucket with 256 millions of objects. Now my problem is that bucket syncing seems to be stopped or stalled. I am %100 certain of which but I failed to figure out the exact problem from "radosgw-admin bucket sync status". It is quite possible that I maybe still don't know the proper tool to figure out the root cause. From the RGW logs I could capture this: "check_bucket_shards: resharding needed: stats.num_objects=256166901 shard max_objects=75000000". Cluster ceph.conf includes "rgw override bucket index max shards = 750". I suppose I need to reshard this bucket to continue syncing. And the only way to reshard is that I have to stop all RGW services of both clusters, delete the secondary zone, reshard the bucket and resync the bucket from the beginning. Or for a better solution I should divide this large bucket into smaller buckets. This might have no easy way but to migrate with some kind of S3 sync tool (rather a fast one!).

1 1

Proper solution of slow_ops
by Michal Strnad 11 Feb '21

11 Feb '21

Hi all, we are looking for a proper solution of slow_ops. When the disk failed, node is restated ... a lot of slow operations appear. Even if disk (OSD) or node is back again most of slow_ops are still there. On the internet we found only advice that we have to restart monitor. But this is not right approach. Do you have some better solution? How did you treat slow_ops in your production clusters? We are running the latest nautilus on all clusters. Thank you Cheers Michal

3 3

db_devices doesn't show up in exported osd service spec
by Tony Liu 11 Feb '21

11 Feb '21

Hi, When build cluster Octopus 15.2.5 initially, here is the OSD service spec file applied. ``` service_type: osd service_id: osd-spec placement: host_pattern: ceph-osd-[1-3] data_devices: rotational: 1 db_devices: rotational: 0 ``` After applying it, all HDDs were added and DB of each hdd is created on SSD. Here is the export of OSD service spec. ``` # ceph orch ls --service_name osd.osd-spec --export service_type: osd service_id: osd-spec service_name: osd.osd-spec placement: host_pattern: ceph-osd-[1-3] spec: data_devices: rotational: 1 filter_logic: AND objectstore: bluestore ``` Why db_devices doesn't show up there? When I replace a disk recently, when the new disk was installed and zapped, OSD was automatically re-created, but DB was created on HDD, not SSD. I assume this is because of that missing db_devices? I tried to update service spec, the same result, db_devices doesn't show up when export it. Is this some known issue or something I am missing? Thanks! Tony

5 18

RGW/Swift 404 error when listing/deleting a newly created empty bucket
by Mike Cave 11 Feb '21

11 Feb '21

So, as the subject states I have an issue with buckets returning a 404 error when they are listed immediately after being created; as well the bucket fails to be deleted if you try to delete it immediately after creation. The behaviour is intermittent. If I leave the bucket in place for a few minutes, the bucket behaves normally. I’m thinking this is a metadata issue or something along those lines but I’m out of my depth now. To the best of our knowledge the cluster has not changed in any way since the same tests were run in December with no errors. We are running Ceph 14.2.16 on all parts of the cluster. I am using the python-swift client for the connection on a CentOS7 machine. Can replicate the results from the mons or an external client as well. I’m willing to share my test script as well if you would like to see how I’m generating the error. Here is a piece of the logs in case I missed something in the interpretation (log level at 20): 14:23:17.069 7faba00df700 1 ====== starting new request req=0x55fb7a138700 ===== 14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id = tx000000000000000000094-0060245cd5-2b8949-default 14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7 14:23:17.069 7faba00df700 10 host=<NameRemoved> 14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 14:23:17.069 7faba00df700 -1 res_query() failed 14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/404test 14:23:17.069 7faba00df700 10 ver=v1 first=404test req= 14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket scheduling with dmclock client=3 cost=1 14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying requester 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::TempURLEngine denied with reason=-13 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket swift_user=xmcc:swift 14:23:17.069 7faba00df700 20 build_token token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::SignedTokenEngine granted access 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket normalizing buckets and tenants 14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init permissions 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+root+404test : hit (negative entry) 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket recalculating target 14:23:17.069 7faba00df700 10 Starting retarget 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket reading permissions 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op mask 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket required_mask= 4 user.op_mask=7 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op permissions 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- Getting permissions begin with perm_mask=50 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50 14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc 14:23:17.069 7faba00df700 5 Found permission: 15 14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op params 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket pre-executing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket executing 14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: bucket 404test not found 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket completing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op status=-2002 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http status=404 14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op status=-2002 http_status=404 latency=0s ====== -- Mike Cave I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day.

1 0

krbd crc
by Seena Fallah 11 Feb '21

11 Feb '21

Hi, I have a few questions about krbd on kernel 4.15 1. Does it support msgr v2? (If not which kernel supports msgr v2?) 2. If krbd is using msgr v1, does it checksum (CRC) the messages that it sends to see for example if the write is correct or not? and if it does checksums, If there were a problem in write how does it react to that? For example, does it raise I/O Error or retry or...? Thanks.

3 4

how far can we go using vstart.sh script for fake dev cluster-HELP
by Bobby 11 Feb '21

11 Feb '21

Hi, Ceph source code contains a script called vstart.sh which allows developers to quickly test their code using a simple deployment on your development system. Here: https://docs.ceph.com/en/latest//dev/quick_guide/ I am really curious that how far we can go with vstart.sh script. While my development cluster is running, I use tools like rados bench, rbd and rbd-nbd to benchmark simple workload and test my code. Do we have options to change the network settings in the fake cluster built from vstart script and later benchmark it? For example , trying 1gbit ethernet and 10gbit ethernet. Thanks

1 0

Re: Dashboard : Block image listing and infos
by Gilles Mocellin 11 Feb '21

11 Feb '21

Hi ! I respond to the list, as it may help others. I also reorder the response. > On Mon, Jan 18, 2021 at 2:41 PM Gilles Mocellin < > > gilles.mocellin(a)nuagelibre.org> wrote: > > Hello Cephers, > > > > On a new cluster, I only have 2 RBD block images, and the Dashboard > > doesn't manage to list them correctly. > > > > I have this message : > > Warning > > Displaying previously cached data for pool veeam-repos. > > > > Sometime it disappears, but as soon as I reload or return to the listing > > page, it's there. > > > > What I've seen, is a high CPU load due to ceph-mgr on the active > > manager. > > And also stack-traces like this : [...] > > dashboard.exceptions.ViewCacheNoDataException: ViewCache: unable to > > retrieve data > > > > I also see that, when I try to edit an image : > > > > 2021-01-18T11:13:26.383+0100 7f00199ca700 0 [dashboard ERROR > > frontend.error] > > (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam-> > repos%252Fveeam-repo2-vol1 > > <https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam-repos% > > 252Fveeam-repo2-vol1>): Cannot read property 'features_name' of > > undefined > > > > TypeError: Cannot read property 'features_name' of undefined [...] > > > > But that's perhaps just becaus I open an Edit window on the image and it > > does not have the datas. > > The Edit window is empty, and I can't edit things, especially, I wan't > > to resize the image. > > [...] > > -- > > Gilles Le jeudi 21 janvier 2021, 21:56:58 CET Ernesto Puerta a écrit : > Hey Gilles, > > If I'm not wrong, that exception (ViewCacheNoDataException) happens when > the dashboard is unable to gather all required data from Ceph within a > defined timeout (5 secs I think, since the UI refreshes the data every ~5 > seconds). > > It'd be great if you could provide the steps to reproduce it and some > insights into your environment (number of RBD pools, number of RBD images, > snapshots, etc.). > > Kind Regards, > > Ernesto OK, As it is now, it always hapens, on the image listing, I have the Warning and the list is not always up to date, if I create an image, I must wait very long to see it. Also, I can not edit the 2 big images I have. Perhaps the size is important, they are 2 images of 40 TB. If I create a 1 GB test image, I can edit and resize it. But impossible withe the big image, the windows opens but all the fields are empty. Also, if it can matter, the images use a data pool (EC 3+2). I have 2 pools, a replicated one for metadatas veeam-repos (replic x3), and a data pool veeam-repos.data (EC 3+2). My cluster has 6 nodes with AMD 16 cores CPU, 128 GB RAM, 10 8 TB HDD. So 60 OSD. Soon doubling everything to 12 nodes. Usage, as the pool and image names can tell, is to mount RBD image as a XFS filesystem for a Veeam Backup Repository (krbd, because nbd-rbd tailed regularly, especially during fstrim).

2 2

Data sync init vs bucket sync init
by Szabo, Istvan (Agoda) 11 Feb '21

11 Feb '21

Hi, What’s the difference between data sync init and bucket sync init? Data initialise the complete cluster? Bucket only bucket? I see when initialise finished, have shards behind but doesn’t do anything with it? What is the proper steps to bring things back to sync? Init Run Restart ?? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Data sync init vs bucket sync init
by Szabo, Istvan (Agoda) 11 Feb '21

11 Feb '21

Hi, What’s the difference between data sync init and bucket sync init? Data initialise the complete cluster? Bucket only bucket? I see when initialise finished, have shards behind but doesn’t do anything with it? What is the proper steps to bring things back to sync? Init Run Restart ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0