January 2021 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

Fwd: ceph-fuse false passed X_OK check
by Alex Taylor 30 Mar '21

30 Mar '21

Hi Patrick, Any updates? Looking forward to your reply :D On Thu, Dec 17, 2020 at 11:39 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote: > > On Wed, Dec 16, 2020 at 5:46 PM Alex Taylor <alexu4993(a)gmail.com> wrote: > > > > Hi Cephers, > > > > I'm using VSCode remote development with a docker server. It worked OK > > but fails to start the debugger after /root mounted by ceph-fuse. The > > log shows that the binary passes access X_OK check but cannot be > > actually executed. see: > > > > ``` > > strace_log: access("/root/.vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7", > > X_OK) = 0 > > > > root@develop:~# ls -alh > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > -rw-r--r-- 1 root root 978 Dec 10 13:06 > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > ``` > > > > I also test the access syscall on ext4, xfs and even cephfs kernel > > client, all of them return -EACCES, which is expected (the extension > > will then explicitly call chmod +x). > > > > After some digging in the code, I found it is probably caused by > > https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L5549-L5550. > > So here come two questions: > > 1. Is this a bug or is there any concern I missed? > > I tried reproducing it with the master branch and could not. It might > be due to an older fuse/ceph. I suggest you upgrade! > I tried the master(332a188d9b3c4eb5c5ad2720b7299913c5a772ee) as well and the issue still exists. My test program is: ``` #include <stdio.h> #include <unistd.h> int main() { int r; const char path[] = "test"; r = access(path, F_OK); printf("file exists: %d\n", r); r = access(path, X_OK); printf("file executable: %d\n", r); return 0; } ``` And the test result: ``` # local filesystem: ext4 root@f626800a6e85:~# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:13 test root@f626800a6e85:~# ./a.out file exists: 0 file executable: -1 root@f626800a6e85:~# findmnt -t fuse.ceph-fuse TARGET SOURCE FSTYPE OPTIONS /root/mnt ceph-fuse fuse.ceph-fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other root@f626800a6e85:~# cd mnt # ceph-fuse root@f626800a6e85:~/mnt# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:10 test root@f626800a6e85:~/mnt# ./a.out file exists: 0 file executable: 0 root@f626800a6e85:~/mnt# ./test bash: ./test: Permission denied ``` Again, ceph-fuse says file `test` is executable but in fact it can't be executed. The kernel version I'm testing on is: ``` root@f626800a6e85:~/mnt# uname -ar Linux f626800a6e85 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux ``` Please try the program above and make sure you're running it as root user, thank you. And if the reproduction still fails, please let me know the kernel version. > > 2. It works again with fuse_default_permissions=true, any drawbacks if > > this option is set? > > Correctness (ironically, for you) and performance. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >

2 1

CephFS max_file_size
by Mark Schouten 25 Mar '21

25 Mar '21

Hi, There is a default limit of 1TiB for the max_file_size in CephFS. I altered that to 2TiB, but I now got a request for storing a file up to 7TiB. I'd expect the limit to be there for a reason, but what is the risk of setting that value to say 10TiB? -- Mark Schouten <mark(a)tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208

5 4

fixing future rctimes
by Dan van der Ster 23 Mar '21

23 Mar '21

Hi all, We have a few subdirs with an rctime in the future. # getfattr -n ceph.dir.rctime session # file: session ceph.dir.rctime="2576387188.090" I can't find any subdir or item in that directory with that rctime, so I presume that there was previously a file and that rctime cannot go backwards [1] Is there any way to fix these rctimes so they show the latest ctime of the subtree? Also -- are we still relying on the client clock to set the rctime / ctime of a file? Would it make sense to limit ctime/rctime for any update to the current time on the MDS ? Best Regards, Dan [1] https://github.com/ceph/ceph/pull/24023/commits/920ef964311a61fcc6c0d6671b7…

2 3

Removing secondary data pool from mds
by Michael Thomas 13 Mar '21

13 Mar '21

I have a cephfs secondary (non-root) data pool with unfound and degraded objects that I have not been able to recover[1]. I created an additional data pool and used "setfattr -n ceph.dir.layout.pool' and a very long rsync to move the files off of the degraded pool and onto the new pool. This has completed, and using find + 'getfattr -n ceph.file.layout.pool', I verified that no files are using the old pool anymore. No ceph.dir.layout.pool attributes point to the old pool either. However, the old pool still reports that there are objects in the old pool, likely the same ones that were unfound/degraded from before: https://pastebin.com/qzVA7eZr Based on a old message from the mailing list[2], I checked the MDS for stray objects (ceph daemon mds.ceph4 dump cache file.txt ; grep -i stray file.txt) and found 36 stray entries in the cache: https://pastebin.com/MHkpw3DV. However, I'm not certain how to map these stray cache objects to clients that may be accessing them. 'rados -p fs.data.archive.frames ls' shows 145 objects. Looking at the parent of each object shows 2 strays: for obj in $(cat rados.ls.txt) ; do echo $obj ; rados -p fs.data.archive.frames getxattr $obj parent | strings ; done [...] 10000020fa1.00000000 10000020fa1 stray6 10000020fbc.00000000 10000020fbc stray6 [...] ...before getting stuck on one object for over 5 minutes (then I gave up): 1000005b1af.00000083 What can I do to make sure this pool is ready to be safely deleted from cephfs (ceph fs rm_data_pool archive fs.data.archive.frames)? --Mike [1]https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QHFOGEKXK7VDNNSKR74BA6IIMGGIXBXA/#7YQ6SSTESM5LTFVLQK3FSYFW5FDXJ5CF [2]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005233.h…

2 5

Remapped PGs
by David Orman 02 Mar '21

02 Mar '21

Hi, We see that we have 5 'remapped' PGs, but are unclear why/what to do about it. We shifted some target ratios for the autobalancer and it resulted in this state. When adjusting ratio, we noticed two OSDs go down, but we just restarted the container for those OSDs with podman, and they came back up. Here's status output: ################### root@ceph01:~# ceph status INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w) mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl, ceph02.gcglcg, ceph04.smbdew, ceph05.yropto osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs data: pools: 3 pools, 1057 pgs objects: 18.00M objects, 69 TiB usage: 119 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 1056 active+clean 1 active+clean+scrubbing+deep io: client: 859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr root@ceph01:~# ################### When I look at ceph pg dump, I don't see any marked as remapped: ################### root@ceph01:~# ceph pg dump |grep remapped INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 dumped all root@ceph01:~# ################### Any idea what might be going on/how to recover? All OSDs are up. Health is 'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman 2.0.3.

2 4

Nautilus Cluster Struggling to Come Back Online
by William Law 26 Feb '21

26 Feb '21

I guess as a sort of follow up from my previous post. Our Nautilus (14.2.16 on ubuntu 18.04) cluster had some sort of event that caused many of the machines to have memory errors. The aftermath is that initially some OSDs had (and continue to have) this error https://tracker.ceph.com/issues/48827 others won't start for various reasons. The OSDs that *will* start are badly behind the current epoch for the most part. It sounds very similar to this: https://blog.noc.grnet.gr/2016/10/18/surviving-a-ceph-cluster-outage-the-ha… We are having trouble getting things back online. I think the path forward is to: -set noup/nodown/noout/nobackfill/and wait for the OSDs that run to come up; we were making good progress yesterday until some of the OSDs crashed with OOM errors. We are again moving forward but understandably nervous. -export the PGs from questionable OSDs and and then rebuild the OSDs; import the PGs if necessary (very likely). Repeat until we are up. Any suggestions for increasing speed? We are using noup/nobackfill/norebalance/pause but the epoch catchup is taking a very long time. Any tips for keeping the epoch from moving forward or speeding up the OSDs catching up? How can we estimate how long it should take? Thank you for any ideas or assistance anyone can provide. Will

2 2

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

Default data pool in CEPH
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi, I have a CEPH 15.2.4 running in a docker. How to configure for use a specific data pool? i try put the follow line in the ceph.conf but the changes not working. . [client.myclient] rbd default data pool = Mydatapool I need it to configure for erasure pool with cloudstack Can help me? , where is the ceph conf we i need configure? Thanks. -- Untitled Document

4 6

Re: NFS Ganesha NFSv3
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi Thanks for the reply. cephadm runs ceph containers automatically. How to set privileged mode in ceph container? -- > El 23/9/20 a las 13:24, Daniel Gryniewicz escribió: >> NFSv3 needs privileges to connect to the portmapper. Try running >> your docker container in privileged mode, and see if that helps. >> >> Daniel >> >> On 9/23/20 11:42 AM, Gabriel Medve wrote: >>> Hi, >>> >>> I have a CEPH 15.2.5 running in a docker , i configure nfs ganesha >>> with nfs version 3 but i can not mount it. >>> If configure ganesha with nfs version 4 i can mounted without >>> problems but i need the version 3 . >>> >>> The error is mount.nfs: Protocol not supported >>> >>> Can help me? >>> >>> Thanks. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- > Untitled Document

2 1

Possible to disable check: x pool(s) have no replicas configured
by Marc Roos 23 Feb '21

23 Feb '21

Is it possible to disable checking on 'x pool(s) have no replicas configured', so I don't have this HEALTH_WARN constantly. Or is there some other disadvantage of keeping some empty 1x replication test pools?

2 1

RBD Image can't be formatted - blk_error
by Gaël THEROND 21 Feb '21

21 Feb '21

Hi everyone! I'm facing a weird issue with one of my CEPH clusters: OS: CentOS - 8.2.2004 (Core) CEPH: Nautilus 14.2.11 - stable RBD using erasure code profile (K=3; m=2) When I want to format one of my RBD image (client side) I've got the following kernel messages multiple time with different sector IDs: *[2417011.790154] blk_update_request: I/O error, dev rbd23, sector 164743869184 op 0x3:(DISCARD) flags 0x4000 phys_seg 1 prio class 0[2417011.791404] rbd: rbd23: discard at objno 20110336 2490368~1703936 result -1 * At first I thought about a faulty disk BUT the monitoring system is not showing anything faulty so I decided to run manual tests on all my OSDs to look at disk health using smartctl etc. None of them is marked as not healthy and actually they don't get any counter with faulty sectors/read or writes and the Wear Level is 99% So, the only particularity of this image is it is a 80Tb image, but it shouldn't be an issue as we already have that kind of image size used on another pool. If anyone have a clue at how I could sort this out, I'll be more than happy ^^ Kind regards!

2 5

OSDs cannot join, MON leader at 100%
by Frank Schilder 17 Feb '21

17 Feb '21

Dear cephers, I was doing some maintenance yesterday involving shutdown-power up cycles of ceph servers. With the last server I run into a problem. The server runs an MDS and a couple of OSDs. After reboot, the MDS joined the MDS cluster without problems, but the OSDs didn't come up. This was 1 out of 12 servers and I had no such problems with the other 11. I also observed that "ceph status" was responding very slow. Upon further inspection, I found out that 2 of my 3 MONs (the leader and a peon) were running at 100% CPU. Client I/O was continuing, probably because the last cluster map remained valid. On our node performance monitoring I could see that the 2 busy MONs were showing extraordinary network activity. This state lasted for over one hour. After the MONs settled down, the OSDs finally joined as well and everything went back to normal. The other instance I have seen similar behaviour was, when I restarted a MON on an empty disk and the re-sync was extremely slow due to a too large value for mon_sync_max_payload_size. This time, I'm pretty sure it was MON-client communication; see below. Are there any settings similar to mon_sync_max_payload_size that could influence responsiveness of MONs in a similar way? Why do I suspect it is MON-client communication? In our monitoring, I do not see the huge amount of packages sent by the MONs arriving at any other ceph daemon. They seem to be distributed over client nodes, but since we have a large count of client nodes (>550) this is covered by the background network traffic. A second clue is that I have had such extended lock-ups before and, whenever I checked, I only observed these in case the leader had a large share of client sessions. For example, yesterday the client session count per MON was: ceph-01: 1339 (leader) ceph-02: 189 (peon) ceph-03: 839 (peon) I usually restart the leader when such a critical distribution occurs. As long as the leader has the fewest client sessions, I never observe this problem. Ceph version is 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable). Thanks for any clues! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 5

CEPHFS - MDS gracefull handover of rank 0
by Martin Hronek 12 Feb '21

12 Feb '21

Hello fellow CEPH-users, currently we are updating our CEPH(14.2.16) and making changes to some config settings. TLDR: is there a way to make a graceful MDS active node shutdown without loosing the caps, open files and client connections? Something like handover active state, promote standby to active, ...? Sadly we run into some difficulties when restarting MDS Nodes. While we had two active nodes and one standby we initially though that this would have a nice handover when restarting the active rank ... sadly we saw how the node was going through the states: replay-reconnect-rejoin-active as nicely visualized here https://docs.ceph.com/en/latest/cephfs/mds-states/ This left some nodes going into timeouts until the standby node has gone into the active state again, most probably since the cephfs hast already some 600k folders and 3M files and from the client side it took more than 30s. So before the next MDS the FS config where changed to one active and one standby-replay node, the idea was that since the MDS replay nodes follows the active one the handover would be smoother. The active state was reached faster, but we still noticed some hiccups on the clients while the new active MDS was waiting for clients to reconnect(state up:reconnect) after the failover. The next idea was to do a manual node promotion, graceful shutdown or something similar - where the open caps and sessions would be handed over ... but I did not find any hint in the docs regarding this functionality. But, this should somehow be possible (imho), since when adding a second active mds node (max_mds 2) and then removing it again (max_mds 1) the rank 1 node goes to stopping-state and hands over all clients/caps to rank 0 without interruptions for the clients. Therefore my question: how can one gracefully shutdown an active rank 0 mds node or promote an standby node to the active state without loosing open files/caps or client sessions? Thanks in advance, M

4 6

Re: Dashboard : Block image listing and infos
by Gilles Mocellin 11 Feb '21

11 Feb '21

Hi ! I respond to the list, as it may help others. I also reorder the response. > On Mon, Jan 18, 2021 at 2:41 PM Gilles Mocellin < > > gilles.mocellin(a)nuagelibre.org> wrote: > > Hello Cephers, > > > > On a new cluster, I only have 2 RBD block images, and the Dashboard > > doesn't manage to list them correctly. > > > > I have this message : > > Warning > > Displaying previously cached data for pool veeam-repos. > > > > Sometime it disappears, but as soon as I reload or return to the listing > > page, it's there. > > > > What I've seen, is a high CPU load due to ceph-mgr on the active > > manager. > > And also stack-traces like this : [...] > > dashboard.exceptions.ViewCacheNoDataException: ViewCache: unable to > > retrieve data > > > > I also see that, when I try to edit an image : > > > > 2021-01-18T11:13:26.383+0100 7f00199ca700 0 [dashboard ERROR > > frontend.error] > > (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam-> > repos%252Fveeam-repo2-vol1 > > <https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam-repos% > > 252Fveeam-repo2-vol1>): Cannot read property 'features_name' of > > undefined > > > > TypeError: Cannot read property 'features_name' of undefined [...] > > > > But that's perhaps just becaus I open an Edit window on the image and it > > does not have the datas. > > The Edit window is empty, and I can't edit things, especially, I wan't > > to resize the image. > > [...] > > -- > > Gilles Le jeudi 21 janvier 2021, 21:56:58 CET Ernesto Puerta a écrit : > Hey Gilles, > > If I'm not wrong, that exception (ViewCacheNoDataException) happens when > the dashboard is unable to gather all required data from Ceph within a > defined timeout (5 secs I think, since the UI refreshes the data every ~5 > seconds). > > It'd be great if you could provide the steps to reproduce it and some > insights into your environment (number of RBD pools, number of RBD images, > snapshots, etc.). > > Kind Regards, > > Ernesto OK, As it is now, it always hapens, on the image listing, I have the Warning and the list is not always up to date, if I create an image, I must wait very long to see it. Also, I can not edit the 2 big images I have. Perhaps the size is important, they are 2 images of 40 TB. If I create a 1 GB test image, I can edit and resize it. But impossible withe the big image, the windows opens but all the fields are empty. Also, if it can matter, the images use a data pool (EC 3+2). I have 2 pools, a replicated one for metadatas veeam-repos (replic x3), and a data pool veeam-repos.data (EC 3+2). My cluster has 6 nodes with AMD 16 cores CPU, 128 GB RAM, 10 8 TB HDD. So 60 OSD. Soon doubling everything to 12 nodes. Usage, as the pool and image names can tell, is to mount RBD image as a XFS filesystem for a Veeam Backup Repository (krbd, because nbd-rbd tailed regularly, especially during fstrim).

2 2

Speed of S3 Ceph gateways
by michal.strnad＠cesnet.cz 09 Feb '21

09 Feb '21

Hi all, We are testing our S3 Ceph endpoints and we are not satisfied with its speed. Our results are something between around 120 - 150 MB/s depending on small/bigger files. This is good for 1Gbps connection, but not for 10GE or more. We've tried the most recent versions of the AWS CLI, s3cmd, s4cmd, s3fs ... programs. Of course we are using multipart upload/download which is precondition for parallel upload/download. Also we tried multi-thread (25 or more threads) transfer in s4cmd but still we don't get proper results. For proof of concept that high speed can be achieved we have written small script in bash which uses multi-part & parallel transfer and can saturate at least 10GE without problem. I would like to ask you, if you know proper program and its parameters, so we can saturate n x 10GE if needed? We are using the latest nautilus. S3 gateways have much more computer power and bandwidth to internet then it is used right now. Thank you Regards Michal Strnad

4 3

Can see objects with "rados ls" but cannot delete them with "rados rm"
by James, GleSYS 09 Feb '21

09 Feb '21

Hi, We have in issue in our cluster (octopus 15.2.7) where we’re unable to remove orphaned objects from a pool, despite the fact these objects can be listed with “rados ls”. Here is an example of an orphaned object which we can list (not sure why multiple objects are returned with the same name…related to the issue perhaps?) rados ls -p default.rgw.buckets.data | grep -i 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 And the error message when we try to stat / rm the object: rados stat -p default.rgw.buckets.data 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 error stat-ing default.rgw.buckets.data/5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6: (2) No such file or directory rados -p default.rgw.buckets.data rm 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6 error removing default.rgw.buckets.data>5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83__shadow_anon_backup_xxxx_xx_xx_090109_7812500.bak.vLHmbxS4DAnRMDVjBYG-5X6iSmepDD6: (2) No such file or directory The bucket with id "5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83” was deleted from radosgw a few months ago, but we still have approximately 450,000 objects with this bucket id that are orphaned: cat orphan-list-202101191211.out | grep -i 5a5c812a-3d31-xxxx-xxxx-xxxxxxxxxxxx.4811659.83 | wc -l 448683 I can also see from our metrics that prior to deletion there was about 10TB of compressed data stored in this bucket, and this has not been reclaimed in the pool usage after the bucket was deleted. Anyone have any suggestions on how we can remove these objects and reclaim the space? We’re not using snapshots or cache tiers in our environment. Thanks, James.

3 3

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Using RBD to pack billions of small files
by Loïc Dachary 03 Feb '21

03 Feb '21

Bonjour, In the context Software Heritage (a noble mission to preserve all source code)[0], artifacts have an average size of ~3KB and there are billions of them. They never change and are never deleted. To save space it would make sense to write them, one after the other, in an every growing RBD volume (more than 100TB). An index, located somewhere else, would record the offset and size of the artifacts in the volume. I wonder if someone already implemented this idea with success? And if not... does anyone see a reason why it would be a bad idea? Cheers [0] https://docs.softwareheritage.org/ -- Loïc Dachary, Artisan Logiciel Libre

8 16

radosgw bucket index issue
by Fox, Kevin M 02 Feb '21

02 Feb '21

We have a fairly old cluster that has over time been upgraded to nautilus. We were digging through some things and found 3 bucket indexes without a corresponding bucket. They should have been deleted but somehow were left behind. When we try and delete the bucket index, it will not allow it as the bucket is not found. The bucket index list command works fine though without the bucket. Is there a way to delete the indexes? Maybe somehow relink the bucket so it can be deleted again? Thanks, Kevin

1 1

osd recommended scheduler
by Andrei Mikhailovsky 02 Feb '21

02 Feb '21

Hello everyone, Could some one please let me know what is the recommended modern kernel disk scheduler that should be used for SSD and HDD osds? The information in the manuals is pretty dated and refer to the schedulers which have been deprecated from the recent kernels. Thanks Andrei

4 5

radosgw process crashes multiple times an hour
by Andrei Mikhailovsky 02 Feb '21

02 Feb '21

Hello, I am experiencing very frequent crashes of the radosgw service. It happens multiple times every hour. As an example, over the last 12 hours we've had 35 crashes. Has anyone experienced similar behaviour of the radosgw octopus release service? More info below: Radosgw service is running on two Ubuntu servers. I have tried upgrading OS on one of the servers to Ubuntu 20.04 with latest updates. The second server is still running Ubuntu 18.04. Both services crash occasionally, but the service which is running on Ubuntu 20.04 crashes far more often it seems. The ceph cluster itself is pretty old and was initially setup around 2013. The cluster was updated pretty regularly with every major release. Currently, I've got Octopus 15.2.8 running on all osd, mon, mgr and radosgw servers. Crash Backtrace: ceph crash info 2021-01-28T11:36:48.912771Z_08f80efd-c0ad-4551-88ce-905ca9cd3aa8 |less { "backtrace": [ "(()+0x46210) [0x7f815a49a210]", "(gsignal()+0xcb) [0x7f815a49a18b]", "(abort()+0x12b) [0x7f815a479859]", "(()+0x9e951) [0x7f8150ee9951]", "(()+0xaa47c) [0x7f8150ef547c]", "(()+0xaa4e7) [0x7f8150ef54e7]", "(()+0xaa799) [0x7f8150ef5799]", "(()+0x344ba) [0x7f815a1404ba]", "(()+0x71e04) [0x7f815a17de04]", "(librados::v14_2_0::IoCtx::nobjects_begin(librados::v14_2_0::ObjectCursor const&, ceph::buffer::v15_2_0::list const&)+0x5d) [0x7f815a18c7bd]", "(RGWSI_RADOS::Pool::List::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWAccessListFilter*)+0x115) [0x7f815b0d9935]", "(RGWSI_SysObj_Core::pool_list_objects_init(rgw_pool const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWSI_SysObj::Pool::ListCtx*)+0x255) [0x7f815abd7035]", "(RGWSI_MetaBackend_SObj::list_init(RGWSI_MetaBackend::Context*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x206) [0x7f815b0ccfe6]", "(RGWMetadataHandler_GenericMetaBE::list_keys_init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void**)+0x41) [0x7f815ad23201]", "(RGWMetadataManager::list_keys_init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void**)+0x71) [0x7f815ad254d1]", "(AsyncMetadataList::_send_request()+0x9b) [0x7f815b13c70b]", "(RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x25) [0x7f815ae60f25]", "(RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0x11) [0x7f815ae69401]", "(ThreadPool::worker(ThreadPool::WorkThread*)+0x5bb) [0x7f81517b072b]", "(ThreadPool::WorkThread::entry()+0x15) [0x7f81517b17f5]", "(()+0x9609) [0x7f815130d609]", "(clone()+0x43) [0x7f815a576293]" ], "ceph_version": "15.2.8", "crash_id": "2021-01-28T11:36:48.912771Z_08f80efd-c0ad-4551-88ce-905ca9cd3aa8", "entity_name": "client.radosgw1.gateway", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "20.04.1 LTS (Focal Fossa)", "os_version_id": "20.04", "process_name": "radosgw", "stack_sig": "347474f09a756104ac2bb99d80e0c1fba3e9dc6f26e4ef68fe55946c103b274a", "timestamp": "2021-01-28T11:36:48.912771Z", "utsname_hostname": "arh-ibstorage1-ib", "utsname_machine": "x86_64", "utsname_release": "5.4.0-64-generic", "utsname_sysname": "Linux", "utsname_version": "#72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021" } radosgw.log file (file names were redacted): -25> 2021-01-28T11:36:48.794+0000 7f8043fff700 1 civetweb: 0x7f814c0cf010: 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT /<file_name>-u115134.JPG HTTP/1.1" 400 460 - - -24> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 ====== starting new request req=0x7f80437f5780 ===== -23> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s initializing for trans_id = tx000000000000000001431-006012a1d0-31197b5c-default -22> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s getting op 1 -21> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj verifying requester -20> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj normalizing buckets and tenants -19> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj init permissions -18> 2021-01-28T11:36:48.814+0000 7f80437fe700 0 req 5169 0s NOTICE: invalid dest placement: default-placement/REDUCED_REDUNDANCY -17> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 op->ERRORHANDLER: err_no=-22 new_err_no=-22 -16> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj op status=0 -15> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj http status=400 -14> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 ====== req done req=0x7f80437f5780 op status=0 http_status=400 latency=0s ====== -13> 2021-01-28T11:36:48.822+0000 7f80437fe700 1 civetweb: 0x7f814c0cf9e8: 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT /<file_name>-d20201223-u115132.JPG HTTP/1.1" 400 460 - - -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request req=0x7f8043ff6780 ===== -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for trans_id = tx000000000000000001432-006012a1d0-31197b5c-default -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1 -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying requester -8> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj normalizing buckets and tenants -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request req=0x7f8043ff6780 ===== -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for trans_id = tx000000000000000001432-006012a1d0-31197b5c-default -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1 -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying requester -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request req=0x7f8043ff6780 ===== -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for trans_id = tx000000000000000001432-006012a1d0-31197b5c-default -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1 -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying requester -8> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj normalizing buckets and tenants -7> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj init permissions -6> 2021-01-28T11:36:48.878+0000 7f8043fff700 0 req 5170 0s NOTICE: invalid dest placement: default-placement/REDUCED_REDUNDANCY -5> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 op->ERRORHANDLER: err_no=-22 new_err_no=-22 -4> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj op status=0 -3> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj http status=400 -2> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== req done req=0x7f8043ff6780 op status=0 http_status=400 latency=0s ====== -1> 2021-01-28T11:36:48.886+0000 7f8043fff700 1 civetweb: 0x7f814c0cf010: 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT /<file_name>-223-u115136.JPG HTTP/1.1" 400 460 - - 0> 2021-01-28T11:36:48.910+0000 7f8128ff9700 -1 *** Caught signal (Aborted) ** 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 30417 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 framework: civetweb 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 framework conf key: port, val: 443s Could someone help me troubleshoot and fix the issue? Thanks Andrei

3 4

Unable to enable RBD-Mirror Snapshot on image when VM is using RBD
by Adam Boyhan 01 Feb '21

01 Feb '21

This is a odd one. I don't hit it all the time so I don't think its expected behavior. Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a KVM VM. Other times I hit the following error, the only way I can get around it is to power down the KVM VM. root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument

2 4

Balancing with upmap
by Francois Legrand 01 Feb '21

01 Feb '21

Hi all, I have a cluster with 116 disks (24 new disks of 16TB added in december and the rest of 8TB) running nautilus 14.2.16. I moved (8 month ago) from crush_compat to upmap balancing. But the cluster seems not well balanced, with a number of pgs on the 8TB disks varying from 26 to 52 ! And an occupation from 35 to 69%. The recent 16 TB disks are more homogeneous with 48 to 61 pgs and space between 30 and 43%. Last week, I realized that some osd were maybe not using upmap because I did a ceph osd crush weight-set ls and got (compat) as result. Thus I ran a ceph osd crush weight-set rm-compat which triggered some rebalancing. Now there is no more recovery for 2 days, but the cluster is still unbalanced. As far as I understand, upmap is supposed to reach an equal number of pgs on all the disks (I guess weighted by their capacity). Thus I would expect more or less 30 pgs on the 8TB disks and 60 on the 16TB and around 50% usage on all. Which is not the case (by far). The problem is that it impact the free available space in the pools (264Ti while there is more than 578Ti free in the cluster) because free space seems to be based on space available before the first osd will be full ! Is it normal ? Did I missed something ? What could I do ? F.

3 11

Multisite recovering shards
by Szabo, Istvan (Agoda) 01 Feb '21

01 Feb '21

Hi, I’ve never seen in our multisite sync status healthy output, almost all the sync shards are recovering. What can I do with recovering shards? We have 1 realm, 1 zonegroup and inside the zonegroup we have 3 zones in 3 different geo location. We are using octopus 15.2.7 for bucket sync with symmetrical replication. The user is at the moment migrating their data and the sites are always behind which is replicated from the place where it was uploaded. I’ve restarted all rgw and disable / enable bucket sync, it started to work, but I think when it comes to close sync it will stop again due to the recovering shards. Any idea? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 4

data on sda with metadata on lvm partition?
by Matt Piermarini 30 Jan '21

30 Jan '21

Hello all - Hi. After running this dev cluster with a single osd (/dev/sda) hdd in each node (6), I want to now put the metadata on the nvme disk which is also used as boot. There is plenty of space left on the nvme, so I re-did the logical volumes to make a 50gb LV for the metadata, thinking I'd put the metdata on thenvme/LV and use the entire /dev/sda as data. Before I really go down this rabbit hole, just want opinions if this is something that should work? I've tried both Cepth 15.2.7 & Ceph 15.2.8, each with different errors. This particular trace is ceph15.2.8. This is under Rook, so rook is doing: <snip> exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --osds-per-device 1 /dev/sda --db-devices /dev/cephDB/database --report provision 2021-01-28 01:46:56.186043 D | exec: --> passed data devices: 1 physical, 0 LVM provision 2021-01-28 01:46:56.186074 D | exec: --> relative data size: 1.0 provision 2021-01-28 01:46:56.186079 D | exec: --> passed block_db devices: 0 physical, 1 LVM provision 2021-01-28 01:46:56.186092 D | exec: provision 2021-01-28 01:46:56.186104 D | exec: Total OSDs: 1 provision 2021-01-28 01:46:56.186107 D | exec: provision 2021-01-28 01:46:56.186111 D | exec: Type Path LV Size % of device provision 2021-01-28 01:46:56.186114 D | exec: ---------------------------------------------------------------------------------------------------- provision 2021-01-28 01:46:56.186117 D | exec: data /dev/sda 3.64 TB 100.00% provision 2021-01-28 01:46:56.186121 D | exec: block_db /dev/cephDB/database 51.65 GB 10000.00%<snip> It fails with the stack trace below, essentially complaining it can't PARTUUID for the LV. exec: Running command: /usr/sbin/lvcreate --yes -l 953861 -n osd-block-2acd94f6-0fed-423b-8540-ae93c0621c2e ceph-b6da5679-543b-4a79-9cc2-e4e308ba61a4 exec: stderr: Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will manage logical volume symlinks in device directory. exec: stderr: Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will obtain device list by scanning device directory. exec: stderr: Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, device-mapper library will manage device nodes in device directory. exec: stdout: Logical volume "osd-block-2acd94f6-0fed-423b-8540-ae93c0621c2e" created. exec: --> blkid could not detect a PARTUUID for device: /dev/cephDB/database exec: --> Was unable to complete a new OSD, will rollback changes exec: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it exec: stderr: purged osd.0 exec: Traceback (most recent call last): exec: File "/usr/sbin/ceph-volume", line 11, in <module> exec: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() exec: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__ exec: self.main(self.argv) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc exec: return f(*a, **kw) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main exec: terminal.dispatch(self.mapper, subcommand_args) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch exec: instance.main() exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main exec: terminal.dispatch(self.mapper, self.argv) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch exec: instance.main() exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root exec: return func(*a, **kw) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 415, in main exec: self._execute(plan) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 431, in _execute exec: p.safe_prepare(argparse.Namespace(**args)) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare exec: self.prepare() exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root exec: return func(*a, **kw) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 382, in prepare exec: self.args.block_db_slots) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 189, in setup_device exec: name_uuid = self.get_ptuuid(device_name) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 135, in get_ptuuid exec: raise RuntimeError('unable to use device') exec: RuntimeError: unable to use device provision failed to configure devices: failed to initialize devices: failed ceph-volume: exit status 1 Thanks for any ideas/help.

1 0

RBD-Mirror Snapshot Backup Image Uses
by Adam Boyhan 29 Jan '21

29 Jan '21

I have been doing some testing with RBD-Mirror Snapshots to a remote Ceph cluster. Does anyone know if the images on the remote cluster can be utilized in anyway? Would love the ability to clone them, or even readonly would be nice.

5 26

Rotating Service Keys
by William Law 29 Jan '21

29 Jan '21

Hi - We keep on getting errors like these on specific OSDs with Nautilus (14.2.16): 2021-01-29 06:14:19.174 7fbeaab92c00 -1 osd.8 12568359 unable to obtain rotating service keys; retrying 2021-01-29 06:14:49.173 7fbeaab92c00 0 monclient: wait_auth_rotating timed out after 30 2021-01-29 06:14:49.173 7fbeaab92c00 -1 osd.8 12568359 unable to obtain rotating service keys; retrying 2021-01-29 06:15:19.173 7fbeaab92c00 0 monclient: wait_auth_rotating timed out after 30 2021-01-29 06:15:19.173 7fbeaab92c00 -1 osd.8 12568359 unable to obtain rotating service keys; retrying 2021-01-29 06:15:49.174 7fbeaab92c00 0 monclient: wait_auth_rotating timed out after 30 2021-01-29 06:15:49.174 7fbeaab92c00 -1 osd.8 12568359 unable to obtain rotating service keys; retrying 2021-01-29 06:15:49.174 7fbeaab92c00 -1 osd.8 12568359 init wait_auth_rotating timed out From googling it seems like it could be a variety of things. We do think time is in sync. It is particularly perplexing as we'll have a single OSD get this error while all other OSDs on the same node are fine. It seems exactly like this: https://tracker.ceph.com/issues/17170 Stopping the managers and restarting the mons fixes it temporarily. From this old thread we do have msgr2 enabled: https://www.spinics.net/lists/ceph-users/msg60631.html This blog seems to point to storage slowness being the root cause in there env: http://www.florentflament.com/blog/ceph-monitor-status-switching-due-to-slo… Any advice for sorting out what is causing this? Thanks, Will

1 0

Unable to use ceph command
by Jens Hyllegaard (Soft Design A/S) 29 Jan '21

29 Jan '21

Hi. I am not sure why this is not working, but I am now unable to use the ceph command on any of my hosts. When I try to launch ceph, I get the following response: [errno 13] RADOS permission denied (error connecting to the cluster) The web management interface is working fine. I have a suspicion that this started after trying to recreate an nfs cluster I first removed the existing one with: ceph nfs cluster delete <id> And then tried to create it again with: ceph nfs cluster create cephfs <id> The command seemed to hang, and after several hours I ended the command with ctrl-c. Since then I have been unable to use the ceph command. This is fortunately a test environment, and it is running Octopus 15.2.8 Does anyone have an idea on how I can get access again? Regards Jens Hyllegaard

3 5

Cannot access "Object Gateway" in dashboard after setting rgw api keys
by Troels Hansen 29 Jan '21

29 Jan '21

Hi, I have tried to enable RGW management in the dashboard. The dashboard works fine, and I tried to add a new system user: radosgw-admin user create --uid=some-user --display-name="User for dashboard" --system and set the accesskey and secret: ceph dashboard set-rgw-api-access-key access-key ceph dashboard set-rgw-api-secret-key secret-key However, after adding these I get this in the dashboard, when I try to access the Object Gateway: "key data not in dict [u'user1', u'user2', u'some-user']" If I try to verify that the credentials are correct with: ceph dashboard get-rgw-api-access-key ceph dashboard get-rgw-api-secret-key secret-key and compare then with the key and secret from: radosgw-admin user info --uid=some-user they match. Running Ceph 14.2.16 To me it looks like it cannot find the user, or match the credentials, but I cannot see what I have done wrong? -- Med venlig hilsen *Troels Hansen* Senior Linux Konsulent Tlf.: 22 43 71 57 tha(a)miracle.dk www.miracle.dk

1 0

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 29 Jan '21

29 Jan '21

On Fri, Jan 29, 2021 at 9:18 AM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > What do I need to do if I want to switch to using a different exchange on > the RabbitMQ endpoint? Or change the amqp-ack-level option that was used? > Would you expect the same problem again? Will the existing connections to > the RabbitMQ server be cleanly terminated? > > I think that changing the ack level would take effect on the next publish (as this is not a feature of the connection, but the calling code), but to change the exchange (or any other parameter of the connection itself), or even creating a new topic to the same endpoint with a different exchange, you would need a restart :-( (tracking this here: https://tracker.ceph.com/issues/46127) > I tried the topic example > <https://www.rabbitmq.com/tutorials/tutorial-five-python.html> from the > RabbitMQ tutorial and I actually got the same behaviour as with Ceph: > messages sent before the consumer queue is attached are lost. From what I > understand this is a *feature* of this type of exchange. See also this > <https://stackoverflow.com/questions/6148381/rabbitmq-persistent-message-wit…> stackoverflow > post. > > this is interesting. and looks like a better model than what we currently have. we should declare our own fanout "gateway" exchange, connected to an "eat all" queue. the users may then connect the consumers directly to it, or via a topic exchange they declare. that would actually fix 2 issues: - durability of messages before clients are connected - exchange name configuration issues > Best, > > Tom > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 28 Jan 2021, at 18:16, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Thu, Jan 28, 2021 at 7:34 PM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> Together with Tom Byrne I ran some more tests today while keeping an eye >> on the logs as well. >> >> We immediately noticed that the nodes were logging errors when uploading >> files like: >> >> 2021-01-28 16:10:45.825 7f56ff5cf700 1 ====== starting new request req=0x7f56ff5c87f0 ===== >> 2021-01-28 16:10:45.828 7f5721e14700 1 AMQP connect: exchange mismatch >> 2021-01-28 16:10:45.828 7f5721e14700 1 ERROR: failed to create push endpoint: amqp://<username>:<password>@<my.rabbitmq.server>:5672 due to: pubsub endpoint configuration error: AMQP: failed to create connection to: amqp://<username>:<password>@<my.rabbitmq.server>:5672 >> 2021-01-28 16:10:45.828 7f571ee0e700 1 ====== req done req=0x7f571ee077f0 op status=0 http_status=200 latency=0.0569997s ====== >> >> >> Which resulted in no connections being established to the RabbitMQ server. >> >> Tom restarted then the Ceph services on one gateway node, which led to >> events being sent to RabbitMQ without blocking, but only if this particular >> node was picked up by the boto3 upload request in the round-robin DNS. >> >> Restarting the Ceph service on all nodes fixed the problem and I got a >> nice steady stream of events to my consumer Python script! >> >> > we should fix it. no restart should be needed if one of the connection > parameters was wrong > > > >> I did notice that any events that were sent while my consumer script was >> not running are lost, as they are not picked up after I restart the script. >> Any thoughts on this? >> >> > this is strange. in our code [1] we don't require immediate transfer of > messages. > how is the exchange declared? > can you check if this is happening when you send messages from a python > producer as well? > > [1] https://github.com/ceph/ceph/blob/master/src/rgw/rgw_amqp.cc#L575 > > > >> Many thanks!! >> >> Best, >> >> Tom >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 27 Jan 2021, at 16:21, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> >> On Wed, Jan 27, 2021 at 5:34 PM Schoonjans, Tom (RFI,RAL,-) < >> Tom.Schoonjans(a)rfi.ac.uk> wrote: >> >>> Looks like there’s already a ticket open for AMQP SSL support: >>> https://tracker.ceph.com/issues/42902 (you opened it ;-)) >>> >>> I will give a try myself if I have some time, but don’t hold your breath >>> with lockdown and home schooling. Also I am not much of a C++ coder. >>> >>> I need to go over the logs with Tom Byrne to see why it is not working >>> properly. And perhaps I will be able to come up with a fix then. >>> >>> However this is what I have run into so far today: >>> >>> 1. After configuring a bucket with a topic using the non-SSL port, I >>> tried a couple of uploads to this bucket. They all hanged, which seemed >>> like something was very wrong, so I Ctrl-C’ed every time. After some time I >>> figured out from the RabbitMQ admin UI that Ceph was indeed connecting to >>> it, and the connections remained so I killed them from the UI. >>> >> >> sending the notification to the rabbitmq server is synchronous with the >> upload to the bucket. so, if the server is slow or not acking the >> notification, the upload request would hang. not that the upload itself is >> done first, but the reply to the client does not happen until rabbitmq >> server acks. >> >> would be great if you can share the radosgw logs. >> maybe the issue is related to the user/password method we use? we use: >> AMQP_SASL_METHOD_PLAIN >> >> one possible workaround would be to set "amqp-ack-level" to "none". in >> this case the radosgw does not wait for an ack >> >> in "pacific" you could use "persistent topics" where the notifications >> are sent asynchronously to the endpoint. >> >> 2. I then wrote a python script with Pika to consume the events, hoping >>> that would stop the blocking. I had some minor success with this. Usually >>> the first three or four uploaded files would generate events that I could >>> consume with my script. >>> >> >> the radosgw is waiting for an ack from the broker, not the end consumer, >> so this should not have mattered... >> did you actually see any notifications delivered to the consumer? >> >> >>> However, the rest would block for ever. I repeated this a couple of >>> times but always the same result. I noticed that after I stopped uploading, >>> removed the bucket and the topic, the connection from Ceph in the RabbitMQ >>> UI remained. I killed it but it came back seconds later from another port >>> on the Ceph cluster. I ended up playing whack-a-mole with this until no >>> more connections would be established from Ceph to RabbitMQ. I probably >>> killed a 100 or so of them. >>> >> >> once you remove the bucket there cannot be new notification sent. if you >> create the bucket again you may see notifications again (this is fixed in >> "pacific"). >> either way, even if the connection to the rabbitmq server would still be >> open, but no new notifications should be sent there. just having the >> connection should not be an issue but would be nice to fix that as well: >> https://tracker.ceph.com/issues/49033 >> >> 3. After this I couldn’t get any events sent anymore. There is no more >>> blocking when uploading, files get written but nothing else happens. No >>> connections are made anymore from Ceph to RabbitMQ. >>> >>> Hope this helps… >>> >> >> yes, this is very helpful! >> >> >>> Best, >>> >>> Tom >>> >>> >>> >>> >>> Dr Tom Schoonjans >>> >>> Research Software Engineer - HPC and Cloud >>> >>> Rosalind Franklin Institute >>> Harwell Science & Innovation Campus >>> Didcot >>> Oxfordshire >>> OX11 0FA >>> United Kingdom >>> >>> https://www.rfi.ac.uk >>> >>> The Rosalind Franklin Institute is a registered charity in England and >>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>> and Wales, No.11266143. Funded by UK Research and Innovation through >>> the Engineering and Physical Sciences Research Council. >>> >>> On 27 Jan 2021, at 13:04, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>> >>> >>> >>> On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < >>> Tom.Schoonjans(a)rfi.ac.uk> wrote: >>> >>>> Hi Yuval, >>>> >>>> >>>> Switching to non-SSL connections to RabbitMQ allowed us to get things >>>> working, although currently it’s not very reliable. >>>> >>> >>> can you please add more about that? what reliability issues did you see? >>> >>> >>>> I will open a new ticket over this if we can’t fix things ourselves. >>>> >>>> >>> this would be great. we have ssl support for kafka and http endpoint, >>> so, if you decide to give it a try you can look at them as examples. >>> and let me know if you have questions or need help. >>> >>> >>> >>>> I will open an issue on the tracker as soon as my account request has >>>> been approved :-) >>>> >>>> Best, >>>> >>>> Tom >>>> >>>> >>>> >>>> >>>> >>>> Dr Tom Schoonjans >>>> >>>> Research Software Engineer - HPC and Cloud >>>> >>>> Rosalind Franklin Institute >>>> Harwell Science & Innovation Campus >>>> Didcot >>>> Oxfordshire >>>> OX11 0FA >>>> United Kingdom >>>> >>>> https://www.rfi.ac.uk >>>> >>>> The Rosalind Franklin Institute is a registered charity in England and >>>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>>> and Wales, No.11266143. Funded by UK Research and Innovation through >>>> the Engineering and Physical Sciences Research Council. >>>> >>>> On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>>> >>>> >>>> >>>> On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < >>>> Tom.Schoonjans(a)rfi.ac.uk> wrote: >>>> >>>>> Hi Yuval, >>>>> >>>>> >>>>> I worked on this earlier today with Tom Byrne and I think I may be >>>>> able to provide some more information. >>>>> >>>>> I set up the RabbitMQ server myself, and created the exchange with >>>>> type ’topic’ before configuring the bucket. >>>>> >>>>> Not sure if this matters, but the RabbitMQ endpoint is reached over >>>>> SSL, using certificates generated with Letsencrypt. >>>>> >>>>> >>>> it actually does. we don't support amqp over ssl. >>>> feel free to open a tracker for that - as we should probably support >>>> that! >>>> but note that it would probably be backported only to later versions >>>> than nautilus. >>>> >>>> >>>> >>>>> Many thanks, >>>>> >>>>> Tom >>>>> >>>>> >>>>> >>>>> Dr Tom Schoonjans >>>>> >>>>> Research Software Engineer - HPC and Cloud >>>>> >>>>> Rosalind Franklin Institute >>>>> Harwell Science & Innovation Campus >>>>> Didcot >>>>> Oxfordshire >>>>> OX11 0FA >>>>> United Kingdom >>>>> >>>>> https://www.rfi.ac.uk >>>>> >>>>> The Rosalind Franklin Institute is a registered charity in England and >>>>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>>>> and Wales, No.11266143. Funded by UK Research and Innovation through >>>>> the Engineering and Physical Sciences Research Council. >>>>> >>>>> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>>>> >>>>> Hi Tom, >>>>> Did you create the exchange in rabbitmq? The RGW does not create it >>>>> and assume it is already created? >>>>> Could you increase the log level in RGW and see if there are more log >>>>> messages that have "AMQP" in them? >>>>> >>>>> Thanks, >>>>> >>>>> Yuval >>>>> >>>>> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >>>>> tom.byrne(a)stfc.ac.uk> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We've been trying to get RGW Bucket notifications working with a >>>>>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>>>>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>>>>> a message to the endpoint, the message never appears in the queue, and we >>>>>> get this error from in the RGW logs: >>>>>> >>>>>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>>>>> Endpoint >>>>>> URI: amqp://user:pass@host:5671 >>>>>> Topic: ceph-topic-test >>>>>> Exchange: ceph-test >>>>>> Ack Level: broker failed, with error: -4098 >>>>>> >>>>>> We've confirmed the URI is correct, and that the gateway host can >>>>>> send messages to the RabbitMQ via a standalone script (using the same >>>>>> information as in the URI). Does anyone have any hints about how to dig >>>>>> into this? >>>>>> >>>>>> Cheers, >>>>>> Tom >>>>>> >>>>>> This email and any attachments are intended solely for the use of the >>>>>> named recipients. If you are not the intended recipient you must not use, >>>>>> disclose, copy or distribute this email or any of its attachments and >>>>>> should notify the sender immediately and delete this email from your >>>>>> system. UK Research and Innovation (UKRI) has taken every reasonable >>>>>> precaution to minimise risk of this email or any attachments containing >>>>>> viruses or malware but the recipient should carry out its own virus and >>>>>> malware checks before opening the attachments. UKRI does not accept any >>>>>> liability for any losses or damages which the recipient may sustain due to >>>>>> presence of any viruses. Opinions, conclusions or other information in this >>>>>> message and attachments that are not related directly to UKRI business are >>>>>> solely those of the author and do not represent the views of UKRI. >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> >

1 0

Differences betwen heap stats and dump dump_mempools
by 展荣臻（信泰） 29 Jan '21

29 Jan '21

Hello,all "ceph daemon osd.0 heap stats" or ceph daemon osd.0 dump_mempools" analyse memory of osd used. The "Virtual address space used" of "ceph daemon osd.0 heap stats" output is lager than total.bytes of "ceph daemon osd.0 dump_mempools"output. I dig into source code and I find the memory of "ceph daemon osd.0 heap stats" output is allocated by malloc. Is it contain memory of dump_mempools output? what differences betwen them? thanks for any reply.

1 0

scrub errors: inconsistent PGs
by Void Star Nill 29 Jan '21

29 Jan '21

Hello all, One of our clusters running nautilus release 14.2.15 is reporting health error. It reports that there are inconsistent PGs. However, when I inspect each of the reported PGs, I dont see any inconsistencies. Any inputs on what's going on? $ sudo ceph health detail HEALTH_ERR 3 scrub errors; Possible data damage: 3 pgs inconsistent OSD_SCRUB_ERRORS 3 scrub errors PG_DAMAGED Possible data damage: 3 pgs inconsistent pg 2.a4 is active+clean+inconsistent, acting [2,60,73] pg 2.2b3 is active+clean+inconsistent, acting [15,3,38] pg 2.758 is active+clean+inconsistent, acting [4,40,35] $ rados list-inconsistent-obj 2.758 --format=json-pretty { "epoch": 9211, "inconsistents": [] } $ rados list-inconsistent-obj 2.a4 --format=json-pretty { "epoch": 9213, "inconsistents": [] } $ rados list-inconsistent-obj 2.758 --format=json-pretty { "epoch": 9211, "inconsistents": [] } Regards, Shridhar

2 1

14.2.16 Low space hindering backfill after reboot
by Marco Pizzolo 28 Jan '21

28 Jan '21

Hello Everyone, We seem to be having a problem on one of our ceph clusters post the OS patch and reboot of one of the nodes. The three other nodes are showing OSD fill rates of 77%-81%, but the 60 OSDs contained in the host that was just rebooted are varying between 64% and 90% since the reboot occurred. The three other nodes have not yet been patched or rebooted. The result is: health: HEALTH_WARN 15 nearfull osd(s) 7 pool(s) nearfull Low space hindering backfill (add storage if this doesn't resolve itself): 15 pgs backfill_toofull Degraded data redundancy: 170940/1437684990 objects degraded (0.012%), 4 pgs degraded, 4 pgs undersized services: mon: 3 daemons, quorum prdceph01,prdceph02,prdceph03 (age 6h) mgr: prdceph01(active, since 5w), standbys: prdceph02, prdceph03, prdceph04 mds: ArchiveRepository:1 {0=prdceph01=up:active} 3 up:standby osd: 240 osds: 240 up (since 6h), 240 in (since 27h); 16 remapped pgs task status: scrub status: mds.prdceph01: idle data: pools: 7 pools, 8384 pgs objects: 479.23M objects, 557 TiB usage: 1.7 PiB used, 454 TiB / 2.1 PiB avail pgs: 170940/1437684990 objects degraded (0.012%) 4155186/1437684990 objects misplaced (0.289%) 8332 active+clean 36 active+clean+scrubbing+deep 11 active+remapped+backfill_toofull 2 active+undersized+degraded+remapped+backfill_toofull 2 active+forced_recovery+undersized+degraded+remapped+forced_backfill+backfill_toofull 1 active+remapped+backfilling io: client: 9.6 MiB/s rd, 820 KiB/s wr, 1.02k op/s rd, 189 op/s wr recovery: 0 B/s, 25 keys/s, 10 objects/s Any suggestions would be greatly appreciated, as currently it is not able to complete the repair, nor will it backfill, even when attempting to force. Many thanks in advance. Marco

2 2

extended multisite downtime...
by Christopher Durham 28 Jan '21

28 Jan '21

Hi, There is a potential that my Ceph RGW multi site soluton may be down for an extended time (2 weeks?) for a physical relocation. Some questions, particularly in regard to RGW 1. Is there any limit on downtime after which I might have to restart an entire sync? I want to still be able to write data during the downtime to the remaining 2. How do I change the master zone and/or zonegroup? My cluster is a 2-site config with one zonegroup (so I won't have to change that, wth 2 zones. If I am physically moving the current master, how do I change the master so I can continue to do bucket meta data ops (new buckets, etc) while the old master site is in transition? And also: 3. My site is mostly RGW, but also has some Cephfs and rbd. Ay issues with those? I am not replicating across sites for those.4. How do I re-IP said cluster if need be? Thanks for any info. -Chris

1 0

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 28 Jan '21

28 Jan '21

On Thu, Jan 28, 2021 at 7:34 PM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > Together with Tom Byrne I ran some more tests today while keeping an eye > on the logs as well. > > We immediately noticed that the nodes were logging errors when uploading > files like: > > 2021-01-28 16:10:45.825 7f56ff5cf700 1 ====== starting new request req=0x7f56ff5c87f0 ===== > 2021-01-28 16:10:45.828 7f5721e14700 1 AMQP connect: exchange mismatch > 2021-01-28 16:10:45.828 7f5721e14700 1 ERROR: failed to create push endpoint: amqp://<username>:<password>@<my.rabbitmq.server>:5672 due to: pubsub endpoint configuration error: AMQP: failed to create connection to: amqp://<username>:<password>@<my.rabbitmq.server>:5672 > 2021-01-28 16:10:45.828 7f571ee0e700 1 ====== req done req=0x7f571ee077f0 op status=0 http_status=200 latency=0.0569997s ====== > > > Which resulted in no connections being established to the RabbitMQ server. > > Tom restarted then the Ceph services on one gateway node, which led to > events being sent to RabbitMQ without blocking, but only if this particular > node was picked up by the boto3 upload request in the round-robin DNS. > > Restarting the Ceph service on all nodes fixed the problem and I got a > nice steady stream of events to my consumer Python script! > > we should fix it. no restart should be needed if one of the connection parameters was wrong > I did notice that any events that were sent while my consumer script was > not running are lost, as they are not picked up after I restart the script. > Any thoughts on this? > > this is strange. in our code [1] we don't require immediate transfer of messages. how is the exchange declared? can you check if this is happening when you send messages from a python producer as well? [1] https://github.com/ceph/ceph/blob/master/src/rgw/rgw_amqp.cc#L575 > Many thanks!! > > Best, > > Tom > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 27 Jan 2021, at 16:21, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > On Wed, Jan 27, 2021 at 5:34 PM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Looks like there’s already a ticket open for AMQP SSL support: >> https://tracker.ceph.com/issues/42902 (you opened it ;-)) >> >> I will give a try myself if I have some time, but don’t hold your breath >> with lockdown and home schooling. Also I am not much of a C++ coder. >> >> I need to go over the logs with Tom Byrne to see why it is not working >> properly. And perhaps I will be able to come up with a fix then. >> >> However this is what I have run into so far today: >> >> 1. After configuring a bucket with a topic using the non-SSL port, I >> tried a couple of uploads to this bucket. They all hanged, which seemed >> like something was very wrong, so I Ctrl-C’ed every time. After some time I >> figured out from the RabbitMQ admin UI that Ceph was indeed connecting to >> it, and the connections remained so I killed them from the UI. >> > > sending the notification to the rabbitmq server is synchronous with the > upload to the bucket. so, if the server is slow or not acking the > notification, the upload request would hang. not that the upload itself is > done first, but the reply to the client does not happen until rabbitmq > server acks. > > would be great if you can share the radosgw logs. > maybe the issue is related to the user/password method we use? we use: > AMQP_SASL_METHOD_PLAIN > > one possible workaround would be to set "amqp-ack-level" to "none". in > this case the radosgw does not wait for an ack > > in "pacific" you could use "persistent topics" where the notifications are > sent asynchronously to the endpoint. > > 2. I then wrote a python script with Pika to consume the events, hoping >> that would stop the blocking. I had some minor success with this. Usually >> the first three or four uploaded files would generate events that I could >> consume with my script. >> > > the radosgw is waiting for an ack from the broker, not the end consumer, > so this should not have mattered... > did you actually see any notifications delivered to the consumer? > > >> However, the rest would block for ever. I repeated this a couple of times >> but always the same result. I noticed that after I stopped uploading, >> removed the bucket and the topic, the connection from Ceph in the RabbitMQ >> UI remained. I killed it but it came back seconds later from another port >> on the Ceph cluster. I ended up playing whack-a-mole with this until no >> more connections would be established from Ceph to RabbitMQ. I probably >> killed a 100 or so of them. >> > > once you remove the bucket there cannot be new notification sent. if you > create the bucket again you may see notifications again (this is fixed in > "pacific"). > either way, even if the connection to the rabbitmq server would still be > open, but no new notifications should be sent there. just having the > connection should not be an issue but would be nice to fix that as well: > https://tracker.ceph.com/issues/49033 > > 3. After this I couldn’t get any events sent anymore. There is no more >> blocking when uploading, files get written but nothing else happens. No >> connections are made anymore from Ceph to RabbitMQ. >> >> Hope this helps… >> > > yes, this is very helpful! > > >> Best, >> >> Tom >> >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 27 Jan 2021, at 13:04, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> >> >> On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < >> Tom.Schoonjans(a)rfi.ac.uk> wrote: >> >>> Hi Yuval, >>> >>> >>> Switching to non-SSL connections to RabbitMQ allowed us to get things >>> working, although currently it’s not very reliable. >>> >> >> can you please add more about that? what reliability issues did you see? >> >> >>> I will open a new ticket over this if we can’t fix things ourselves. >>> >>> >> this would be great. we have ssl support for kafka and http endpoint, so, >> if you decide to give it a try you can look at them as examples. >> and let me know if you have questions or need help. >> >> >> >>> I will open an issue on the tracker as soon as my account request has >>> been approved :-) >>> >>> Best, >>> >>> Tom >>> >>> >>> >>> >>> >>> Dr Tom Schoonjans >>> >>> Research Software Engineer - HPC and Cloud >>> >>> Rosalind Franklin Institute >>> Harwell Science & Innovation Campus >>> Didcot >>> Oxfordshire >>> OX11 0FA >>> United Kingdom >>> >>> https://www.rfi.ac.uk >>> >>> The Rosalind Franklin Institute is a registered charity in England and >>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>> and Wales, No.11266143. Funded by UK Research and Innovation through >>> the Engineering and Physical Sciences Research Council. >>> >>> On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>> >>> >>> >>> On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < >>> Tom.Schoonjans(a)rfi.ac.uk> wrote: >>> >>>> Hi Yuval, >>>> >>>> >>>> I worked on this earlier today with Tom Byrne and I think I may be able >>>> to provide some more information. >>>> >>>> I set up the RabbitMQ server myself, and created the exchange with type >>>> ’topic’ before configuring the bucket. >>>> >>>> Not sure if this matters, but the RabbitMQ endpoint is reached over >>>> SSL, using certificates generated with Letsencrypt. >>>> >>>> >>> it actually does. we don't support amqp over ssl. >>> feel free to open a tracker for that - as we should probably support >>> that! >>> but note that it would probably be backported only to later versions >>> than nautilus. >>> >>> >>> >>>> Many thanks, >>>> >>>> Tom >>>> >>>> >>>> >>>> Dr Tom Schoonjans >>>> >>>> Research Software Engineer - HPC and Cloud >>>> >>>> Rosalind Franklin Institute >>>> Harwell Science & Innovation Campus >>>> Didcot >>>> Oxfordshire >>>> OX11 0FA >>>> United Kingdom >>>> >>>> https://www.rfi.ac.uk >>>> >>>> The Rosalind Franklin Institute is a registered charity in England and >>>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>>> and Wales, No.11266143. Funded by UK Research and Innovation through >>>> the Engineering and Physical Sciences Research Council. >>>> >>>> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>>> >>>> Hi Tom, >>>> Did you create the exchange in rabbitmq? The RGW does not create it and >>>> assume it is already created? >>>> Could you increase the log level in RGW and see if there are more log >>>> messages that have "AMQP" in them? >>>> >>>> Thanks, >>>> >>>> Yuval >>>> >>>> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >>>> tom.byrne(a)stfc.ac.uk> wrote: >>>> >>>>> Hi all, >>>>> >>>>> We've been trying to get RGW Bucket notifications working with a >>>>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>>>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>>>> a message to the endpoint, the message never appears in the queue, and we >>>>> get this error from in the RGW logs: >>>>> >>>>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>>>> Endpoint >>>>> URI: amqp://user:pass@host:5671 >>>>> Topic: ceph-topic-test >>>>> Exchange: ceph-test >>>>> Ack Level: broker failed, with error: -4098 >>>>> >>>>> We've confirmed the URI is correct, and that the gateway host can send >>>>> messages to the RabbitMQ via a standalone script (using the same >>>>> information as in the URI). Does anyone have any hints about how to dig >>>>> into this? >>>>> >>>>> Cheers, >>>>> Tom >>>>> >>>>> This email and any attachments are intended solely for the use of the >>>>> named recipients. If you are not the intended recipient you must not use, >>>>> disclose, copy or distribute this email or any of its attachments and >>>>> should notify the sender immediately and delete this email from your >>>>> system. UK Research and Innovation (UKRI) has taken every reasonable >>>>> precaution to minimise risk of this email or any attachments containing >>>>> viruses or malware but the recipient should carry out its own virus and >>>>> malware checks before opening the attachments. UKRI does not accept any >>>>> liability for any losses or damages which the recipient may sustain due to >>>>> presence of any viruses. Opinions, conclusions or other information in this >>>>> message and attachments that are not related directly to UKRI business are >>>>> solely those of the author and do not represent the views of UKRI. >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> >>>> >>> >> >

1 0

Where has my capacity gone?
by George Yil 28 Jan '21

28 Jan '21

Hi, I have a ceph nautilus (14.2.9) cluster with 10 nodes. Each node has 19x16TB disks attached. I created radosgw pools. secondaryzone.rgw.buckets.data pool is configured as EC 8+2 (jerasure). ceph df shows 2.1PiB MAX AVAIL space. Then I configured radosgw as a secondary zone and 100TiB of S3 data is replicated. But weirdly enough ceph df shows 1.8PiB MAX AVAIL for the same pool. But there is only 100TiB of written data. ceph df also confirms it. I can not figure out where 200TiB capacity is gone. Would someone please tell me what I am missing? Thanks.

2 6

Rbd pool shows 458GB USED but the image is empty
by Szabo, Istvan (Agoda) 28 Jan '21

28 Jan '21

Hi, We have a pool where the user has 2 image. They cleaned up the images, no snaphot in it, but when I see ceph df detail it still shows 458GB in the first column. Why? Thanks ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 5

RGW multi-site sudden accumulation of inconsistent PGs
by Eugen Block 28 Jan '21

28 Jan '21

Hi *, is there any correlation between multi-site clusters and inconsistent PGs? One customer has two Octopus clusters (fresh install a few months ago) which have been expanded recently with new disks. Before that they had one single occurence of inconsistent PGs during a deep-scrub which could be fixed. Since the expansion they get them on a regular basis, something like two times a day or so (I don't have access). The latest occurence was reported as omap_digest mismatch which could be fixed with the pg repair command. There are no signs of failing disks or anything suspicious. And when searching this list I noticed a couple of other threads (one in example is [1]) with this problem, they all had rgw multi-site in common. Is it just a conincidence? Does anyone have some insight or more experience on this? Thanks! Eugen [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-April/034155.html

1 0

Re: Where has my capacity gone?
by George Yil 28 Jan '21

28 Jan '21

Hi Marc, Thanks for participating. At first I thought this is an incorrect report and maybe I need to upgrade to for a bugfix. But I couldn’t find a such a report and I asked here. When people shared experiences it appears there may be two causes. Unbalanced OSDs or Storage Amplification. As far as I understand, this is most likely to be storage amplification. Unbalancing seems to less relevant since this is a fresh cluster. Or I might be misinterpreting ceph osd df. https://pastebin.ubuntu.com/p/ZmQZsGYpr7/ <https://pastebin.ubuntu.com/p/7C9zpXYntR/> So I am trying to figure out the best way to change bluestore_min_alloc_size_hdd. And also I think pool compression can be a quick solution for the future data writes, but I am not %100 sure. Any idea is more than welcome. > On 28 Jan 2021, at 12:29, Marc Roos <Marc(a)f1-outsourcing.eu> wrote: > > Hi George, > > Sorry for asking maybe I skipped an email. But what eventually caused the 'incorrect' report on available storage. > > >

1 0

Planning: Ceph User Survey 2020
by Mike Perez 28 Jan '21

28 Jan '21

Hi everyone, The Ceph User Survey 2020 is being planned by our working group. Please review the draft survey pdf, and let's discuss any changes. You may also join us in the next meeting *on November 25th at 12pm *PT https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group https://tracker.ceph.com/attachments/download/5260/Ceph%20User%20Survey%202… We're aiming to have something ready by mid-December. -- Mike Perez he/him Ceph / Rook / RDO / Gluster Community Architect Open-Source Program Office (OSPO) M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

5 6

Re: PG inconsistent with empty inconsistent objects
by Richard Bade 27 Jan '21

27 Jan '21

Hi Everyone, I also have seen this inconsistent with empty when you do list-inconsistent-obj $ sudo ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1 pgs not deep-scrubbed in time OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21] PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811 $ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty { "epoch": 183807, "inconsistents": [] } Usually these are caused by read errors on the disks, but I've checked all osd hosts that are part of this osd and there's no smart or dmesg errors. Rich ------------------------------ > > Date: Sun, 17 Jan 2021 14:00:01 +0330 > From: Seena Fallah <seenafallah(a)gmail.com> > Subject: [ceph-users] Re: PG inconsistent with empty inconsistent > objects > To: "Alexander E. Patrakov" <patrakov(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > <CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > It's for a long time ago and I don't have the `ceph health detail` output! > > On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov <patrakov(a)gmail.com> > wrote: > > > For a start, please post the "ceph health detail" output. > > > > сб, 19 дек. 2020 г. в 23:48, Seena Fallah <seenafallah(a)gmail.com>: > > > > > > Hi, > > > > > > I'm facing something strange! One of the PGs in my pool got inconsistent > > > and when I run `rados list-inconsistent-obj $PG_ID --format=json-pretty` > > > the `inconsistents` key was empty! What is this? Is it a bug in Ceph > > or..? > > > > > > Thanks. > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > -- > > Alexander E. Patrakov > > CV: http://u.pc.cd/wT8otalK > > >

4 5

radosgw not working - upgraded from mimic to octopus
by Youzhong Yang 27 Jan '21

27 Jan '21

I upgraded our ceph cluster (6 bare metal nodes, 3 rgw VMs) from v13.2.4 to v15.2.8. The mon, mgr, mds and osd daemons were all upgraded successfully, everything looked good. After the radosgw was upgraded, they refused to work, the log messages are at the end of this e-mail. Here are the things I tried: 1. I moved aside the pools for the rgw service, started from scratch (creating realm, zonegroup, zone, users), but when I tried to run 'radosgw-admin user create ...', it appeared to be stuck and never returned, other command like 'radosgw-admin period update --commit' also got stuck. 2. I rolled back radosgw to the old version v13.2.4, then everything works great again. What am I missing here? Is there anything extra that needs to be done for rgw after upgrading from mimic to octopus? Please kindly help. Thanks. --------------------- 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 898 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework: civetweb 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: port, val: 80 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:24:10.192-0500 7f638f79f9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:29:10.195-0500 7f638cbcd700 -1 Initialization timeout, failed to initialize 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 1541 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework: civetweb 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: port, val: 80 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:29:25.883-0500 7f4c213ba9c0 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2021-01-24T09:29:25.883-0500 7f4c213ba9c0 0 ERROR: failed to distribute cache for coredumps.rgw.log:meta.history 2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 978 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework: civetweb 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: port, val: 80 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 0 ERROR: failed to distribute cache for coredumps.rgw.log:meta.history

1 1

Re: Balancing with upmap
by Francois Legrand 27 Jan '21

27 Jan '21

Nope ! Le 27/01/2021 à 17:40, Anthony D'Atri a écrit : > Do you have any override reweights set to values less than 1.0? > > The REWEIGHT column when you run `ceph osd df` > >> On Jan 27, 2021, at 8:15 AM, Francois Legrand <fleg(a)lpnhe.in2p3.fr> wrote: >> >> Hi all, >> I have a cluster with 116 disks (24 new disks of 16TB added in december and the rest of 8TB) running nautilus 14.2.16. >> I moved (8 month ago) from crush_compat to upmap balancing. >> But the cluster seems not well balanced, with a number of pgs on the 8TB disks varying from 26 to 52 ! And an occupation from 35 to 69%. >> The recent 16 TB disks are more homogeneous with 48 to 61 pgs and space between 30 and 43%. >> Last week, I realized that some osd were maybe not using upmap because I did a ceph osd crush weight-set ls and got (compat) as result. >> Thus I ran a ceph osd crush weight-set rm-compat which triggered some rebalancing. Now there is no more recovery for 2 days, but the cluster is still unbalanced. >> As far as I understand, upmap is supposed to reach an equal number of pgs on all the disks (I guess weighted by their capacity). >> Thus I would expect more or less 30 pgs on the 8TB disks and 60 on the 16TB and around 50% usage on all. Which is not the case (by far). >> The problem is that it impact the free available space in the pools (264Ti while there is more than 578Ti free in the cluster) because free space seems to be based on space available before the first osd will be full ! >> Is it normal ? Did I missed something ? What could I do ? >> >> F. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 27 Jan '21

27 Jan '21

On Wed, Jan 27, 2021 at 5:34 PM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Looks like there’s already a ticket open for AMQP SSL support: > https://tracker.ceph.com/issues/42902 (you opened it ;-)) > > I will give a try myself if I have some time, but don’t hold your breath > with lockdown and home schooling. Also I am not much of a C++ coder. > > I need to go over the logs with Tom Byrne to see why it is not working > properly. And perhaps I will be able to come up with a fix then. > > However this is what I have run into so far today: > > 1. After configuring a bucket with a topic using the non-SSL port, I tried > a couple of uploads to this bucket. They all hanged, which seemed like > something was very wrong, so I Ctrl-C’ed every time. After some time I > figured out from the RabbitMQ admin UI that Ceph was indeed connecting to > it, and the connections remained so I killed them from the UI. > sending the notification to the rabbitmq server is synchronous with the upload to the bucket. so, if the server is slow or not acking the notification, the upload request would hang. not that the upload itself is done first, but the reply to the client does not happen until rabbitmq server acks. would be great if you can share the radosgw logs. maybe the issue is related to the user/password method we use? we use: AMQP_SASL_METHOD_PLAIN one possible workaround would be to set "amqp-ack-level" to "none". in this case the radosgw does not wait for an ack in "pacific" you could use "persistent topics" where the notifications are sent asynchronously to the endpoint. 2. I then wrote a python script with Pika to consume the events, hoping > that would stop the blocking. I had some minor success with this. Usually > the first three or four uploaded files would generate events that I could > consume with my script. > the radosgw is waiting for an ack from the broker, not the end consumer, so this should not have mattered... did you actually see any notifications delivered to the consumer? > However, the rest would block for ever. I repeated this a couple of times > but always the same result. I noticed that after I stopped uploading, > removed the bucket and the topic, the connection from Ceph in the RabbitMQ > UI remained. I killed it but it came back seconds later from another port > on the Ceph cluster. I ended up playing whack-a-mole with this until no > more connections would be established from Ceph to RabbitMQ. I probably > killed a 100 or so of them. > once you remove the bucket there cannot be new notification sent. if you create the bucket again you may see notifications again (this is fixed in "pacific"). either way, even if the connection to the rabbitmq server would still be open, but no new notifications should be sent there. just having the connection should not be an issue but would be nice to fix that as well: https://tracker.ceph.com/issues/49033 3. After this I couldn’t get any events sent anymore. There is no more > blocking when uploading, files get written but nothing else happens. No > connections are made anymore from Ceph to RabbitMQ. > > Hope this helps… > yes, this is very helpful! > Best, > > Tom > > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 27 Jan 2021, at 13:04, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> Switching to non-SSL connections to RabbitMQ allowed us to get things >> working, although currently it’s not very reliable. >> > > can you please add more about that? what reliability issues did you see? > > >> I will open a new ticket over this if we can’t fix things ourselves. >> >> > this would be great. we have ssl support for kafka and http endpoint, so, > if you decide to give it a try you can look at them as examples. > and let me know if you have questions or need help. > > > >> I will open an issue on the tracker as soon as my account request has >> been approved :-) >> >> Best, >> >> Tom >> >> >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> >> >> On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < >> Tom.Schoonjans(a)rfi.ac.uk> wrote: >> >>> Hi Yuval, >>> >>> >>> I worked on this earlier today with Tom Byrne and I think I may be able >>> to provide some more information. >>> >>> I set up the RabbitMQ server myself, and created the exchange with type >>> ’topic’ before configuring the bucket. >>> >>> Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, >>> using certificates generated with Letsencrypt. >>> >>> >> it actually does. we don't support amqp over ssl. >> feel free to open a tracker for that - as we should probably support that! >> but note that it would probably be backported only to later versions than >> nautilus. >> >> >> >>> Many thanks, >>> >>> Tom >>> >>> >>> >>> Dr Tom Schoonjans >>> >>> Research Software Engineer - HPC and Cloud >>> >>> Rosalind Franklin Institute >>> Harwell Science & Innovation Campus >>> Didcot >>> Oxfordshire >>> OX11 0FA >>> United Kingdom >>> >>> https://www.rfi.ac.uk >>> >>> The Rosalind Franklin Institute is a registered charity in England and >>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>> and Wales, No.11266143. Funded by UK Research and Innovation through >>> the Engineering and Physical Sciences Research Council. >>> >>> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>> >>> Hi Tom, >>> Did you create the exchange in rabbitmq? The RGW does not create it and >>> assume it is already created? >>> Could you increase the log level in RGW and see if there are more log >>> messages that have "AMQP" in them? >>> >>> Thanks, >>> >>> Yuval >>> >>> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >>> tom.byrne(a)stfc.ac.uk> wrote: >>> >>>> Hi all, >>>> >>>> We've been trying to get RGW Bucket notifications working with a >>>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>>> a message to the endpoint, the message never appears in the queue, and we >>>> get this error from in the RGW logs: >>>> >>>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>>> Endpoint >>>> URI: amqp://user:pass@host:5671 >>>> Topic: ceph-topic-test >>>> Exchange: ceph-test >>>> Ack Level: broker failed, with error: -4098 >>>> >>>> We've confirmed the URI is correct, and that the gateway host can send >>>> messages to the RabbitMQ via a standalone script (using the same >>>> information as in the URI). Does anyone have any hints about how to dig >>>> into this? >>>> >>>> Cheers, >>>> Tom >>>> >>>> This email and any attachments are intended solely for the use of the >>>> named recipients. If you are not the intended recipient you must not use, >>>> disclose, copy or distribute this email or any of its attachments and >>>> should notify the sender immediately and delete this email from your >>>> system. UK Research and Innovation (UKRI) has taken every reasonable >>>> precaution to minimise risk of this email or any attachments containing >>>> viruses or malware but the recipient should carry out its own virus and >>>> malware checks before opening the attachments. UKRI does not accept any >>>> liability for any losses or damages which the recipient may sustain due to >>>> presence of any viruses. Opinions, conclusions or other information in this >>>> message and attachments that are not related directly to UKRI business are >>>> solely those of the author and do not represent the views of UKRI. >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>> >> >

1 0

"ceph orch restart mgr" command creates mgr restart loop
by Chris Read 27 Jan '21

27 Jan '21

Greetings all... I'm busy testing out Ceph and have hit this troublesome bug while following the steps outlined here: https://docs.ceph.com/en/octopus/cephadm/monitoring/#configuring-ssl-tls-fo… When I issue the "ceph orch restart mgr" command, it appears the command is not cleared from a message queue somewhere (I'm still very unclear on many ceph specifics), and so each time the mgr process returns from restart it picks up the message again and keeps restarting itself forever (so far it's been stuck in this state for 45 minutes). Watching the logs we see this going on: $ ceph log last cephadm -w root@ceph-poc-000:~# ceph log last cephadm -w cluster: id: d23bc326-543a-11eb-bfe0-b324db228b6c health: HEALTH_OK services: mon: 5 daemons, quorum ceph-poc-000,ceph-poc-003,ceph-poc-004,ceph-poc-002,ceph-poc-001 (age 2h) mgr: ceph-poc-000.himivo(active, since 4s), standbys: ceph-poc-001.unjulx osd: 10 osds: 10 up (since 2h), 10 in (since 2h) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 10 GiB used, 5.4 TiB / 5.5 TiB avail pgs: 1 active+clean 2021-01-11T20:46:32.976606+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:32.980749+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:33.061519+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:39.156420+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:39.160618+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:39.242603+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:45.299953+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:45.304006+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:45.733495+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:51.871903+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:51.877107+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:51.976190+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:58.000720+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:58.006843+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:58.097163+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:47:04.188630+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:47:04.193501+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:47:04.285509+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:47:10.348099+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:47:10.352340+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:47:10.752243+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available And in the logs for the mgr instance itself we see it keep replaying the message over and over: $ docker logs -f ceph-d23bc326-543a-11eb-bfe0-b324db228b6c-mgr.ceph-poc-000.himivo debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 set uid:gid to 167:167 (ceph:ceph) debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process ceph-mgr, pid 1 debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 pidfile_write: ignore empty --pid-file debug 2021-01-11T20:47:31.414+0000 7f48b0d0d200 1 mgr[py] Loading python module 'alerts' debug 2021-01-11T20:47:31.486+0000 7f48b0d0d200 1 mgr[py] Loading python module 'balancer' debug 2021-01-11T20:47:31.542+0000 7f48b0d0d200 1 mgr[py] Loading python module 'cephadm' debug 2021-01-11T20:47:31.742+0000 7f48b0d0d200 1 mgr[py] Loading python module 'crash' debug 2021-01-11T20:47:31.798+0000 7f48b0d0d200 1 mgr[py] Loading python module 'dashboard' debug 2021-01-11T20:47:32.258+0000 7f48b0d0d200 1 mgr[py] Loading python module 'devicehealth' debug 2021-01-11T20:47:32.306+0000 7f48b0d0d200 1 mgr[py] Loading python module 'diskprediction_local' debug 2021-01-11T20:47:32.498+0000 7f48b0d0d200 1 mgr[py] Loading python module 'influx' debug 2021-01-11T20:47:32.550+0000 7f48b0d0d200 1 mgr[py] Loading python module 'insights' debug 2021-01-11T20:47:32.598+0000 7f48b0d0d200 1 mgr[py] Loading python module 'iostat' debug 2021-01-11T20:47:32.642+0000 7f48b0d0d200 1 mgr[py] Loading python module 'k8sevents' debug 2021-01-11T20:47:33.034+0000 7f48b0d0d200 1 mgr[py] Loading python module 'localpool' debug 2021-01-11T20:47:33.082+0000 7f48b0d0d200 1 mgr[py] Loading python module 'orchestrator' debug 2021-01-11T20:47:33.266+0000 7f48b0d0d200 1 mgr[py] Loading python module 'osd_support' debug 2021-01-11T20:47:33.310+0000 7f48b0d0d200 1 mgr[py] Loading python module 'pg_autoscaler' debug 2021-01-11T20:47:33.382+0000 7f48b0d0d200 1 mgr[py] Loading python module 'progress' debug 2021-01-11T20:47:33.442+0000 7f48b0d0d200 1 mgr[py] Loading python module 'prometheus' debug 2021-01-11T20:47:33.794+0000 7f48b0d0d200 1 mgr[py] Loading python module 'rbd_support' debug 2021-01-11T20:47:33.870+0000 7f48b0d0d200 1 mgr[py] Loading python module 'restful' debug 2021-01-11T20:47:34.086+0000 7f48b0d0d200 1 mgr[py] Loading python module 'rook' debug 2021-01-11T20:47:34.606+0000 7f48b0d0d200 1 mgr[py] Loading python module 'selftest' debug 2021-01-11T20:47:34.654+0000 7f48b0d0d200 1 mgr[py] Loading python module 'status' debug 2021-01-11T20:47:34.710+0000 7f48b0d0d200 1 mgr[py] Loading python module 'telegraf' debug 2021-01-11T20:47:34.762+0000 7f48b0d0d200 1 mgr[py] Loading python module 'telemetry' debug 2021-01-11T20:47:35.034+0000 7f48b0d0d200 1 mgr[py] Loading python module 'test_orchestrator' debug 2021-01-11T20:47:35.146+0000 7f48b0d0d200 1 mgr[py] Loading python module 'volumes' debug 2021-01-11T20:47:35.294+0000 7f48b0d0d200 1 mgr[py] Loading python module 'zabbix' debug 2021-01-11T20:47:35.366+0000 7f489df86700 0 ms_deliver_dispatch: unhandled message 0x55d57b71a420 mon_map magic: 0 v1 from mon.0 v2: 10.128.18.16:3300/0 debug 2021-01-11T20:47:35.758+0000 7f489df86700 1 mgr handle_mgr_map Activating! debug 2021-01-11T20:47:35.758+0000 7f489df86700 1 mgr handle_mgr_map I am now activating debug 2021-01-11T20:47:35.770+0000 7f484ca14700 0 [balancer DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.770+0000 7f484ca14700 1 mgr load Constructed class from module: balancer debug 2021-01-11T20:47:35.774+0000 7f484ca14700 0 [cephadm DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.874+0000 7f484ca14700 1 mgr load Constructed class from module: cephadm debug 2021-01-11T20:47:35.874+0000 7f484ca14700 0 [crash DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.874+0000 7f484ca14700 1 mgr load Constructed class from module: crash debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [dashboard DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: dashboard debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: devicehealth debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [iostat DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: iostat debug 2021-01-11T20:47:35.886+0000 7f484ca14700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.886+0000 7f484ca14700 1 mgr load Constructed class from module: orchestrator debug 2021-01-11T20:47:35.890+0000 7f484ca14700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.890+0000 7f484ca14700 1 mgr load Constructed class from module: pg_autoscaler debug 2021-01-11T20:47:35.890+0000 7f484ca14700 0 [progress DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.890+0000 7f484ca14700 1 mgr load Constructed class from module: progress debug 2021-01-11T20:47:35.898+0000 7f484ca14700 0 [prometheus DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.910+0000 7f484ca14700 1 mgr load Constructed class from module: prometheus debug 2021-01-11T20:47:35.914+0000 7f484ca14700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5) [11/Jan/2021:20:47:35] ENGINE Bus STARTING Warning: Permanently added 'ceph-poc-000,10.128.18.16' (ECDSA) to the list of known hosts. Warning: Permanently added 'ceph-poc-002,10.128.19.248' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.974+0000 7f484ca14700 1 mgr load Constructed class from module: rbd_support Warning: Permanently added 'ceph-poc-001,10.128.28.223' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [restful DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: restful debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [status DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: status debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [telemetry DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: telemetry debug 2021-01-11T20:47:35.982+0000 7f486f3cc700 0 [restful WARNING root] server not running: no certificate configured debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [volumes DEBUG root] setting log level based on debug_mgr: WARNING (1/5) Warning: Permanently added 'ceph-poc-004,10.128.31.60' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.994+0000 7f484ca14700 1 mgr load Constructed class from module: volumes Warning: Permanently added 'ceph-poc-003,10.128.30.127' (ECDSA) to the list of known hosts. CherryPy Checker: The Application mounted at '' has an empty config. [11/Jan/2021:20:47:36] ENGINE Serving on http://:::9283 [11/Jan/2021:20:47:36] ENGINE Bus STARTED debug 2021-01-11T20:47:36.778+0000 7f484d215700 0 log_channel(audit) log [DBG] : from='client.19140 -' entity='client.admin' cmd=[{"prefix": "orch", "action": "restart", "service_name": "mgr", "target": ["mon-mgr", ""]}]: dispatch debug 2021-01-11T20:47:36.778+0000 7f484ca14700 0 log_channel(cephadm) log [INF] : Restart service mgr debug 2021-01-11T20:47:36.806+0000 7f484d215700 0 log_channel(cluster) log [DBG] : pgmap v3: 1 pgs: 1 active+clean; 0 B data, 208 MiB used, 5.4 TiB / 5.5 TiB avail debug 2021-01-11T20:47:37.406+0000 7f489ef88700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 debug 2021-01-11T20:47:37.406+0000 7f489ef88700 -1 mgr handle_mgr_signal *** Got signal Terminated *** Is there anyway to manually clear this stuck message? Will it eventually timeout by itself? Either way it's a pretty unpleasant surprise to find when following the docs so early in the process. Thanks, Chris

2 1

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 27 Jan '21

27 Jan '21

On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > Switching to non-SSL connections to RabbitMQ allowed us to get things > working, although currently it’s not very reliable. > can you please add more about that? what reliability issues did you see? > I will open a new ticket over this if we can’t fix things ourselves. > > this would be great. we have ssl support for kafka and http endpoint, so, if you decide to give it a try you can look at them as examples. and let me know if you have questions or need help. > I will open an issue on the tracker as soon as my account request has been > approved :-) > > Best, > > Tom > > > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> I worked on this earlier today with Tom Byrne and I think I may be able >> to provide some more information. >> >> I set up the RabbitMQ server myself, and created the exchange with type >> ’topic’ before configuring the bucket. >> >> Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, >> using certificates generated with Letsencrypt. >> >> > it actually does. we don't support amqp over ssl. > feel free to open a tracker for that - as we should probably support that! > but note that it would probably be backported only to later versions than > nautilus. > > > >> Many thanks, >> >> Tom >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> Hi Tom, >> Did you create the exchange in rabbitmq? The RGW does not create it and >> assume it is already created? >> Could you increase the log level in RGW and see if there are more log >> messages that have "AMQP" in them? >> >> Thanks, >> >> Yuval >> >> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >> tom.byrne(a)stfc.ac.uk> wrote: >> >>> Hi all, >>> >>> We've been trying to get RGW Bucket notifications working with a >>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>> a message to the endpoint, the message never appears in the queue, and we >>> get this error from in the RGW logs: >>> >>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>> Endpoint >>> URI: amqp://user:pass@host:5671 >>> Topic: ceph-topic-test >>> Exchange: ceph-test >>> Ack Level: broker failed, with error: -4098 >>> >>> We've confirmed the URI is correct, and that the gateway host can send >>> messages to the RabbitMQ via a standalone script (using the same >>> information as in the URI). Does anyone have any hints about how to dig >>> into this? >>> >>> Cheers, >>> Tom >>> >>> This email and any attachments are intended solely for the use of the >>> named recipients. If you are not the intended recipient you must not use, >>> disclose, copy or distribute this email or any of its attachments and >>> should notify the sender immediately and delete this email from your >>> system. UK Research and Innovation (UKRI) has taken every reasonable >>> precaution to minimise risk of this email or any attachments containing >>> viruses or malware but the recipient should carry out its own virus and >>> malware checks before opening the attachments. UKRI does not accept any >>> liability for any losses or damages which the recipient may sustain due to >>> presence of any viruses. Opinions, conclusions or other information in this >>> message and attachments that are not related directly to UKRI business are >>> solely those of the author and do not represent the views of UKRI. >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >> >

1 0

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 26 Jan '21

26 Jan '21

On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > I worked on this earlier today with Tom Byrne and I think I may be able to > provide some more information. > > I set up the RabbitMQ server myself, and created the exchange with type > ’topic’ before configuring the bucket. > > Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, > using certificates generated with Letsencrypt. > > it actually does. we don't support amqp over ssl. feel free to open a tracker for that - as we should probably support that! but note that it would probably be backported only to later versions than nautilus. > Many thanks, > > Tom > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > Hi Tom, > Did you create the exchange in rabbitmq? The RGW does not create it and > assume it is already created? > Could you increase the log level in RGW and see if there are more log > messages that have "AMQP" in them? > > Thanks, > > Yuval > > On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < > tom.byrne(a)stfc.ac.uk> wrote: > >> Hi all, >> >> We've been trying to get RGW Bucket notifications working with a RabbitMQ >> endpoint on our Nautilus 14.2.15 cluster. The gateway host can communicate >> with the rabbitMQ server just fine, but when RGW tries to send a message to >> the endpoint, the message never appears in the queue, and we get this error >> from in the RGW logs: >> >> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >> Endpoint >> URI: amqp://user:pass@host:5671 >> Topic: ceph-topic-test >> Exchange: ceph-test >> Ack Level: broker failed, with error: -4098 >> >> We've confirmed the URI is correct, and that the gateway host can send >> messages to the RabbitMQ via a standalone script (using the same >> information as in the URI). Does anyone have any hints about how to dig >> into this? >> >> Cheers, >> Tom >> >> This email and any attachments are intended solely for the use of the >> named recipients. If you are not the intended recipient you must not use, >> disclose, copy or distribute this email or any of its attachments and >> should notify the sender immediately and delete this email from your >> system. UK Research and Innovation (UKRI) has taken every reasonable >> precaution to minimise risk of this email or any attachments containing >> viruses or malware but the recipient should carry out its own virus and >> malware checks before opening the attachments. UKRI does not accept any >> liability for any losses or damages which the recipient may sustain due to >> presence of any viruses. Opinions, conclusions or other information in this >> message and attachments that are not related directly to UKRI business are >> solely those of the author and do not represent the views of UKRI. >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

1 0

RGW Bucket notification troubleshooting
by Byrne, Thomas (STFC,RAL,SC) 26 Jan '21

26 Jan '21

Hi all, We've been trying to get RGW Bucket notifications working with a RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can communicate with the rabbitMQ server just fine, but when RGW tries to send a message to the endpoint, the message never appears in the queue, and we get this error from in the RGW logs: 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) Endpoint URI: amqp://user:pass@host:5671 Topic: ceph-topic-test Exchange: ceph-test Ack Level: broker failed, with error: -4098 We've confirmed the URI is correct, and that the gateway host can send messages to the RabbitMQ via a standalone script (using the same information as in the URI). Does anyone have any hints about how to dig into this? Cheers, Tom This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

2 1

Re: Where has my capacity gone?
by George Yil 26 Jan '21

26 Jan '21

Sorry for replying late :(. And thanks for the tips. This is a fresh cluster. And I didn’t think data distribution would be a problem. Is this normal? Below is the ceph osd df output. The related pool is hdd only (prod.rgw.buckets.data). I guess there is variance but I couldn’t get the reason. Is it because of PG number which I get help from pg-calculator <https://ceph.io/pgcalc/>? Or is this expected ceph behaviour? #ceph osd df https://pastebin.ubuntu.com/p/ZmQZsGYpr7/ <https://pastebin.ubuntu.com/p/7C9zpXYntR/> I am also sharing related cluster information. Any suggestion would be appreciated. #ceph df https://pastebin.ubuntu.com/p/sXpf99zhnV/ <https://pastebin.ubuntu.com/p/sXpf99zhnV/> #ceph detail df https://pastebin.ubuntu.com/p/dwvwBnnBmv/ <https://pastebin.ubuntu.com/p/dwvwBnnBmv/> #ceph osd pool ls detail https://pastebin.ubuntu.com/p/c2KQD5CGMV/ <https://pastebin.ubuntu.com/p/c2KQD5CGMV/> #crush rules https://pastebin.ubuntu.com/p/X6WsZhV3Zz/ <https://pastebin.ubuntu.com/p/X6WsZhV3Zz/> Thanks. > On 26 Jan 2021, at 11:18, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > ceph osd df | sort -nk8 > >> On Jan 25, 2021, at 11:22 PM, George Yil <georgeyil75(a)gmail.com> wrote: >> >> Hi, >> >> I have a ceph nautilus (14.2.9) cluster with 10 nodes. Each node has >> 19x16TB disks attached. >> >> I created radosgw pools. secondaryzone.rgw.buckets.data pool is configured >> as EC 8+2 (jerasure). >> ceph df shows 2.1PiB MAX AVAIL space. >> >> Then I configured radosgw as a secondary zone and 100TiB of S3 data is >> replicated. >> >> But weirdly enough ceph df shows 1.8PiB MAX AVAIL for the same pool. But >> there is only 100TiB of written data. ceph df also confirms it. I can not >> figure out where 200TiB capacity is gone. >> >> Would someone please tell me what I am missing? >> >> Thanks. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Ceph-mds using a lot of buffer_anon memory
by David Turner 26 Jan '21

26 Jan '21

We just upgraded a cephfs cluster from 12.2.12 to 14.2.11. Our next step is to upgrade to 14.2.16 to troubleshoot this issue, but I thought I'd reach out here first if anyone had any ideas. The clients are still running an older version of ceph-fuse 12.2.4 and it's very difficult to remount all of them. Would probably take a team of us a couple days to restart all of them. I've looked around online and release notes and all of the known memory leaks I've been able to find have been fixed prior to version 14.2.11 so this would be an unknown memory leak. All of the memory is in use in [1] buffer_anon. If left unchecked it will use up over 700GB of memory within 24 hours. On an identical cluster with an equivalent workload still running 12.2.12 [2] buffer_anon information is much healthier. Without any other options or ideas our plan is to upgrade the cluster to 14.2.16 first and then upgrade the clients. Has anyone else come across high buffer_anon usage? [1] "buffer_anon": { "items": 33756758, "bytes": 135025912897 }, [2] "buffer_anon": { "items": 636, "bytes": 273118 },

1 0

libRADOS semantics
by Cary FitzHugh 26 Jan '21

26 Jan '21

are there any good documents on the implicit requirements for librados calls? I can write a file, if it doesn't exist successfully. I can write_full a file, and it works (but many of my files will need to be chunked off disk, not enough RAM) I can append to a nonexistent file and it works. I *cannot* write to a part of an existing file (overwrite a section of it or write past it's current size) --> Errno 95 I *cannot* append to an existing file --> Errno 95 I *cannot* trunc an existing file (nor a non existent file) --> Errno 95 (what is the point of this then??) any ideas / where can one see more detailed semantics of what operations are allowed and when? The librados C API does not really go into any detail about errors and why things happen as they do. This behavior seems similar whether I am admin or a separate user. Thanks for any pointers. Cary

1 1

Auth Questions w/ librados
by Cary FitzHugh 26 Jan '21

26 Jan '21

Hello! I've got a new cluster set up. I created a new user and pool. Creating the user like so: ceph auth add client.myclient mon 'allow r' osd 'allow rw pool=testpool' I have set up librados and python bindings, with a ceph.conf and a keyring file. My trouble is that when I connect with client.admin, things seem to work correctly. but my client.myuser cannot connect.. When I try to connect with my client.myclient I get a RADOS permission denied. cluster = rados.Rados(conffile=conffile.name) My keyring only has one user in it. The conffile only has a [global] section with a mon_host string and the keyring path. I am also unable to connect to the cluster using that user with the ceph CLI client. I have had limited success in debugging any of these situations - all the debugging I might put into the ceph.conf file seems to only work after the client is connected. In testing, I also made the client.myclient user have allow * on all capabilities, and that did nothing that I could see. If anyone has thoughts on where I am stumbling, or how to debug things, that would be much appreciated. It feels like it's a combination of a few small mistakes, but I can't get a handle on the issues. Thank you. Cary

1 1

rados bench error after running vstart script- HELP PLEASE
by Bobby 26 Jan '21

26 Jan '21

Hello, I am having an error while trying to run rados benchmark after running vstart script I run : ../src/vstart.sh -d -n -l and then when I try to run: bin/rados -p rbd bench 30 write it gives me error saying: error opening pool rbd: (2) No such file or directory Can someone please help me in this. I will be grateful Thanks

3 4

Running ceph cluster on different os
by Szabo, Istvan (Agoda) 26 Jan '21

26 Jan '21

Hi, Is there anybody running a cluster with different os? Due to the centos 8 change I might try to add ubuntu osd nodes to centos cluster and decommission the centos slowly but I'm not sure this is possible or not. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 3

Permissions for OSD
by George Shuklin 25 Jan '21

25 Jan '21

Docs for permissions are super vague. What each flag does? What is 'x' permitting? What's the difference between class-write and write? And the last question: can we limit user to reading/writing only to existing objects in the pool? Thanks!

1 0

January Ceph Science Virtual User Group Meeting
by Kevin Hrpcek 25 Jan '21

25 Jan '21

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday January 27th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20210127 <https://pad.ceph.com/p/Ceph_Science_User_Group_20200923> We try to keep it to an hour or less. Ceph calendar event details: January 27th, 2020 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 0

Multisite bucket data inconsistency
by Szabo, Istvan (Agoda) 25 Jan '21

25 Jan '21

Hi, We have bucket sync enabled and seems like it is inconsistent ☹ This is the master zone sync status on that specific bucket: realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) metadata sync no sync (zone is master) data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 126 shards behind shards: [0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-25T11:32:57.726042+0700 [62] 104 shards are recovering recovering shards: [0,2,3,4,5,7,8,9,10,11,12,13,15,16,17,18,19,20,21,22,24,25,26,27,28,29,31,32,33,36,37,38,39,40,42,43,44,45,47,50,51,52,53,54,55,57,58,61,63,65,66,67,68,69,70,71,72,73,74,75,76,78,80,81,82,83,84,85,87,88,90,92,93,95,96,97,98,99,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,123,124,125,126,127] This is the secondary zone where the data has been uploaded: realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 125 shards behind shards: [0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,18,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-25T11:29:32.450031+0700 [61] 126 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,104,105,106,107,108,109,110,111,112,114,115,116,117,118,119,120,121,122,123,124,125,126,127] The pipes are already there: "id": "seo-2", "data_flow": { "symmetrical": [ { "id": "seo-2-flow", "zones": [ "9213182a-14ba-48ad-bde9-289a1c0c0de8", "f20ddd64-924b-4f78-8d2d-dd6c65f98ba9" ] } ] }, "pipes": [ { "id": "seo-2-hkg-ash-pipe", "source": { "bucket": "seo..prerender", "zones": [ "9213182a-14ba-48ad-bde9-289a1c0c0de8" ] }, "dest": { "bucket": "seo..prerender", "zones": [ "f20ddd64-924b-4f78-8d2d-dd6c65f98ba9" ] }, "params": { "source": { "filter": { "tags": [] } }, "dest": {}, "priority": 0, "mode": "system", "user": "" } }, { "id": "seo-2-ash-hkg-pipe", "source": { "bucket": "seo..prerender", "zones": [ "f20ddd64-924b-4f78-8d2d-dd6c65f98ba9" ] }, "dest": { "bucket": "seo..prerender", "zones": [ "9213182a-14ba-48ad-bde9-289a1c0c0de8" ] }, "params": { "source": { "filter": { "tags": [] } }, "dest": {}, "priority": 0, "mode": "system", "user": "" } } ], "status": "enabled" } Any idea to troubleshoot? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 2

cephadm db_slots and wal_slots ignored
by Schweiss, Chip 25 Jan '21

25 Jan '21

I'm trying to set up a new ceph cluster with cephadm on a SUSE SES trial that has Ceph 15.2.8 Each OSD node has 18 rotational SAS disks, 4 NVMe 2TB SSDs for DB, and 2 NVME2 200GB Optane SSDs for WAL. These servers will eventually have 24 rotational SAS disks that they will inherit from existing storage servers. So I don't want all the space used on the DB and WAL SSDs. I suspect from the comment "(db_slots is actually to be favoured here, but it's not implemented yet)" on this page in the docs: https://docs.ceph.com/en/latest/cephadm/drivegroups/#the-advanced-case these parameters are not yet implemented, yet are documented as such under "ADDITIONAL OPTIONS" My osd_spec.yml: service_type: osd service_id: three_tier_osd placement: host_pattern: '*' data_devices: rotational: 1 model: 'ST14000NM0288' db_devices: rotational: 0 model: 'INTEL SSDPE2KX020T8' limit: 6 wal_devices: model: 'INTEL SSDPEL1K200GA' limit: 12 db_slots: 6 wal_slots: 12 All available space is consumed on my DB and WAL SSDs with only 18 OSDs, leaving no room to add additional spindles. Is this still work in progress, or a bug I should report? Possibly related to https://github.com/rook/rook/issues/5026 At the minimum, this appears to be a documentation bug. How can I work around this? -Chip

2 3

radosgw-admin realm pull from the secondary site fails "(13) Permission denied"
by Hayashida, Mami 23 Jan '21

23 Jan '21

I have been trying to create two virtual test clusters to learn about the RGW multisite setting. So far, I have set up two small Nautilus (v.14.2.16) clusters, designated one of them as the "master zone site" and followed every step outlined in the doc ( https://docs.ceph.com/en/nautilus/radosgw/multisite/) including create a system user, updating the period, and restarting the rgw daemon. (For the sake of simplicity, there is only one RGW daemon running on each site.) Once I installed the RGW daemon on the secondary zone site, I tried pulling the realm from the master zone cluster, but ended up with this: ``` $ radosgw-admin realm pull --url=http://<master zone gateway>:80 --access-key=<system_access_key> --secret=<system_secret_key> request failed: (13) Permission denied If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user. ``` I tried adding the --rgw-realm=<realm set up in the primary site>, but the result was the same. I restarted the rgw daemon on both sides -- that did not help, either. The output of all of the following on the master zone side, as far as I could tell, seems correct -- the realm, zonegroup, zone I created are the only ones and set to default. ``` radosgw-admin zone/zonegroup/realm list radosgw-admin zone/zonegroup/realm get ``` On the "master zone" side, the rgw log shows ``` 2021-01-22 13:34:48.404 7fb9ca89e700 1 ====== starting new request req=0x7fb9ca897740 ===== 2021-01-22 13:34:48.428 7fb9ca89e700 1 ====== req done req=0x7fb9ca897740 op status=0 http_status=403 latency=0.0240002s ====== 2021-01-22 13:34:48.428 7fb9ca89e700 1 civetweb: 0x559d6509a000: 10.33.30.55 - - [22/Jan/2021:13:34:48 -0500] "GET /admin/realm HTTP/1.1" 403 318 - - ``` I am using Ubuntu 18.04, Ceph v.14.2.16, deployed using `ceph-deploy`. *Mami Hayashida* *Research Computing Associate* Univ. of Kentucky ITS Research Computing Infrastructure

2 2

Re: Storage down due to MON sync very slow
by Frank Schilder 22 Jan '21

22 Jan '21

Hi Dan, it is possible that the payload reduction also solved or at least reduced a really bad problem that looks related (beware, that's a long one): https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/FBGIJZNFG44… . Since reducing the payload size I still observe these large peaks in the MON network activity. However, it seems that the cluster does not go down like before any more. During these peaks, I see warnings like these: 2021-01-22 12:00:00.000102 [WRN] overall HEALTH_WARN 1 pools nearfull 2021-01-22 11:04:09.156796 [INF] Health check cleared: SLOW_OPS (was: 5 slow ops, oldest one blocked for 75 sec, mon.ceph-02 has slow ops) 2021-01-22 11:04:07.994416 [WRN] Health check update: 5 slow ops, oldest one blocked for 75 sec, mon.ceph-02 has slow ops (SLOW_OPS) 2021-01-22 11:04:01.469498 [WRN] Health check failed: 124 slow ops, oldest one blocked for 82 sec, daemons [mon.ceph-02,mon.ceph-03] have slow ops. (SLOW_OPS) 2021-01-22 11:00:00.000104 [WRN] overall HEALTH_WARN 1 pools nearfull 2021-01-22 10:36:44.576663 [INF] Health check cleared: SLOW_OPS (was: 25 slow ops, oldest one blocked for 42 sec, daemons [mon.ceph-02,mon.ceph-03] have slow ops.) 2021-01-22 10:36:38.543763 [WRN] Health check failed: 18 slow ops, oldest one blocked for 38 sec, daemons [mon.ceph-02,mon.ceph-03] have slow ops. (SLOW_OPS) So, at least stuff is working. I now lean towards the hypothesis that these outages were caused by some synchronisation process between MONs that got less problematic with reducing the payload size. I might be able to reduce my insane beacon time-outs again, but before doing so, do you know of any other communication parameters similar to the mon_sync_max_payload_size that might be relevant in MON-[MON, MGR, OSD] communication? In general, I have the impression that due to such little bugs the recommendation for production clusters should be elevated to at least 5 MONs so that one can afford 2 MONs going out of quorum temporarily. I will upgrade our cluster to 5 MONs as soon as I can. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dan(a)vanderster.com> Sent: 06 January 2021 20:53:14 To: Frank Schilder Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow Yeah I was going to say -- ignore all of the rsync advice in that thread, it is unnecessary. Setting a small mon sync payload works like magic :) -- dan On Wed, Jan 6, 2021 at 8:49 PM Frank Schilder <frans(a)dtu.dk> wrote: > > OK, sorry for all my questions. > > Setting mon_sync_max_payload_size=4096 actually makes the MON sync in no time! Thank you so much :) > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Frank Schilder > Sent: 06 January 2021 20:40:26 > To: Dan van der Ster > Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow > > OK, thanks a lot! I will try it now. Hope the cluster remains responsive. > > I'm wondering about this approach someone brought up in your thread: > > Eventually I stopped one MON, tarballed it's database and used that to > bring back the MON which was upgraded to 13.2.8 > > That work without any hickups. The MON joined again within a few seconds. > > Stopping one MON for a copy would be much shorter storage outage than the sync I'm doing. I guess its the entire mon data directory to copy. I always wondered if this contains data tied to a specific MON. If not, the copy approach could speed things up a lot. What do you think? > > Thanks again and best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Dan van der Ster <dan(a)vanderster.com> > Sent: 06 January 2021 20:36:15 > To: Frank Schilder > Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow > > We have used mon_sync_max_payload_size 4096 on our largest most > important prod cluster since that thread. > The PR from Sage makes something like that the default anyway. (the PR > counts keys rather than bytes, but the effect is the same). > > mon_sync_max_payload_size 4096 should not impact the speed of syncing > -- it simply breaks the sync into smaller more manageable pieces. > (Without this, if you have lots of keys in the mon db, in our case > caused by lots of rbd snapshots, then syncing will never ever > complete). > > -- dan > > On Wed, Jan 6, 2021 at 8:32 PM Frank Schilder <frans(a)dtu.dk> wrote: > > > > Hi Dan, > > > > thanks for that. Will it slow down or accelerate the syncing (will read your post after that e-mail), or will it just allow I/O to continue and sync more in the background? Current value is > > > > mon_sync_max_payload_size 1048576 > > > > Related to that, would building a MON store from OSDs following https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… help providing a head start? Not sure if this procedure works on an active cluster. > > > > Will study your thread now ... > > > > Thanks again and best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: Dan van der Ster <dan(a)vanderster.com> > > Sent: 06 January 2021 20:26:46 > > To: Frank Schilder > > Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow > > > > (obviously just put that config in the ceph.conf on the mons if mimic > > doesn't have ceph config... I don't quite remember.) > > > > -- dan > > > > On Wed, Jan 6, 2021 at 8:25 PM Dan van der Ster <dan(a)vanderster.com> wrote: > > > > > > This sounds a lot like an old thread of mine: > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/M5ZKF7PTEO2… > > > > > > See the discussion about mon_sync_max_payload_size, and the PR that > > > fixed this at some point in nautilus. > > > > > > Our workaround was: > > > > > > ceph config set mon mon_sync_max_payload_size 4096 > > > > > > Hope that helps, > > > > > > Dan > > > > > > > > > On Wed, Jan 6, 2021 at 8:18 PM Frank Schilder <frans(a)dtu.dk> wrote: > > > > > > > > Dear Dan, > > > > > > > > thanks for your fast response. > > > > > > > > Version: mimic 13.2.10. > > > > > > > > Here is the mon_status of the "new" MON during syncing: > > > > > > > > [root@ceph-01 ~]# ceph daemon mon.ceph-01 mon_status > > > > { > > > > "name": "ceph-01", > > > > "rank": 0, > > > > "state": "synchronizing", > > > > "election_epoch": 0, > > > > "quorum": [], > > > > "features": { > > > > "required_con": "144115188346404864", > > > > "required_mon": [ > > > > "kraken", > > > > "luminous", > > > > "mimic", > > > > "osdmap-prune" > > > > ], > > > > "quorum_con": "0", > > > > "quorum_mon": [] > > > > }, > > > > "outside_quorum": [ > > > > "ceph-01" > > > > ], > > > > "extra_probe_peers": [], > > > > "sync_provider": [], > > > > "sync": { > > > > "sync_provider": "mon.2 192.168.32.67:6789/0", > > > > "sync_cookie": 33302773774, > > > > "sync_start_version": 38355711 > > > > }, > > > > "monmap": { > > > > "epoch": 3, > > > > "fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", > > > > "modified": "2019-03-14 23:08:34.717223", > > > > "created": "2019-03-14 22:18:15.088212", > > > > "features": { > > > > "persistent": [ > > > > "kraken", > > > > "luminous", > > > > "mimic", > > > > "osdmap-prune" > > > > ], > > > > "optional": [] > > > > }, > > > > "mons": [ > > > > { > > > > "rank": 0, > > > > "name": "ceph-01", > > > > "addr": "192.168.32.65:6789/0", > > > > "public_addr": "192.168.32.65:6789/0" > > > > }, > > > > { > > > > "rank": 1, > > > > "name": "ceph-02", > > > > "addr": "192.168.32.66:6789/0", > > > > "public_addr": "192.168.32.66:6789/0" > > > > }, > > > > { > > > > "rank": 2, > > > > "name": "ceph-03", > > > > "addr": "192.168.32.67:6789/0", > > > > "public_addr": "192.168.32.67:6789/0" > > > > } > > > > ] > > > > }, > > > > "feature_map": { > > > > "mon": [ > > > > { > > > > "features": "0x3ffddff8ffacfffb", > > > > "release": "luminous", > > > > "num": 1 > > > > } > > > > ], > > > > "mds": [ > > > > { > > > > "features": "0x3ffddff8ffacfffb", > > > > "release": "luminous", > > > > "num": 2 > > > > } > > > > ], > > > > "client": [ > > > > { > > > > "features": "0x2f018fb86aa42ada", > > > > "release": "luminous", > > > > "num": 1 > > > > }, > > > > { > > > > "features": "0x3ffddff8eeacfffb", > > > > "release": "luminous", > > > > "num": 1 > > > > }, > > > > { > > > > "features": "0x3ffddff8ffacfffb", > > > > "release": "luminous", > > > > "num": 17 > > > > } > > > > ] > > > > } > > > > } > > > > > > > > I'm a bit surprised that the other 2 MONs don't remain in quorum until this MON has caught up. Is there any way to monitor the syncing progress? Right now I need to interrupt regularly to allow some I/O, but I have no clue how long I need to wait. > > > > > > > > Thanks for your help! > > > > > > > > Best regards, > > > > ================= > > > > Frank Schilder > > > > AIT Risø Campus > > > > Bygning 109, rum S14 > > > > > > > > ________________________________________ > > > > From: Dan van der Ster <dan(a)vanderster.com> > > > > Sent: 06 January 2021 20:16:44 > > > > To: Frank Schilder > > > > Cc: Ceph Users > > > > Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow > > > > > > > > Which version of Ceph are you running? > > > > > > > > .. dan > > > > > > > > > > > > On Wed, Jan 6, 2021, 8:14 PM Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> wrote: > > > > In the output of the MON I see slow ops warnings: > > > > > > > > debug 2021-01-06 20:12:48.854 7f1a3d29f700 -1 mon.ceph-01@0(synchronizing) e3 get_health_metrics reporting 20 slow ops, oldest is log(1 entries from seq 1 at 2021-01-06 20:00:12.014861) > > > > > > > > There appears to be no progress on this operation, it is stuck. > > > > > > > > Best regards, > > > > ================= > > > > Frank Schilder > > > > AIT Risø Campus > > > > Bygning 109, rum S14 > > > > > > > > ________________________________________ > > > > From: Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> > > > > Sent: 06 January 2021 20:11:25 > > > > To: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > > > > Subject: [ceph-users] Storage down due to MON sync very slow > > > > > > > > Dear all, > > > > > > > > I had to restart one out of 3 MONs on an empty MON DB dir. It is in state syncing right now, but I'm not sure if there is any progress. The cluster is completely unresponsive even though I have 2 healthy MONs. Is there any way to sync the DB directory faster and/or without downtime? > > > > > > > > Thanks a lot! > > > > > > > > Best regards, > > > > ================= > > > > Frank Schilder > > > > AIT Risø Campus > > > > Bygning 109, rum S14 > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Cannot create new OSD node - _read_fsid unparsable uuid
by Ha, Son Hai 22 Jan '21

22 Jan '21

Hello everyone, I'm trying to add an OSD node to my current cluster. I created an lvm volume for this node to use for OSD. My current Ceph version is 14.2.6 and it runs on an RHEL 7 OS. However, I got error when trying to activate the node. I'm confused with the output. I tried to see what really happened, but I don't know how to proceed to pinpoint the issues. I appreciate so much for any help to clear the confusion. * Why did the "mon getmap" command send the result to /dev/stderr? I saw the monmap file having value and it looked like it got the monmap, is there any way to check if the monmap having problem? * What caused "_read_fsid unparsable uuid"? Sincerely, Hai [xxx@xxx.com@xxx430 ~]$ sudo ceph-volume --cluster aap-storage lvm create --data /dev/kubernetes/ceph-osd Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring -i - osd new c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/aap-storage-5 Running command: /sbin/restorecon /var/lib/ceph/osd/aap-storage-5 Running command: /bin/chown -h ceph:ceph /dev/kubernetes/ceph-osd Running command: /bin/chown -R ceph:ceph /dev/dm-9 Running command: /bin/ln -s /dev/kubernetes/ceph-osd /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring mon getmap -o /var/lib/ceph/osd/aap-storage-5/activate.monmap stderr: got monmap epoch 8 Running command: /bin/ceph-authtool /var/lib/ceph/osd/aap-storage-5/keyring --create-keyring --name osd.5 --add-key AQB2RAhgnkcLIBAAdcdR5N4YKzSJmfoA6G6XvA== stdout: creating /var/lib/ceph/osd/aap-storage-5/keyring added entity osd.5 auth(key=AQB2RAhgnkcLIBAAdcdR5N4YKzSJmfoA6G6XvA==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5/ Running command: /bin/ceph-osd --cluster aap-storage --osd-objectstore bluestore --mkfs -i 5 --monmap /var/lib/ceph/osd/aap-storage-5/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/aap-storage-5/ --osd-uuid c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 --setuser ceph --setgroup ceph stderr: 2021-01-20 15:55:52.319 7f0970d3ba80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: kubernetes/ceph-osd Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5 Running command: /bin/ceph-bluestore-tool --cluster=aap-storage prime-osd-dir --dev /dev/kubernetes/ceph-osd --path /var/lib/ceph/osd/aap-storage-5 --no-mon-config Running command: /bin/ln -snf /dev/kubernetes/ceph-osd /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/aap-storage-5/block Running command: /bin/chown -R ceph:ceph /dev/dm-9 Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5 Running command: /bin/systemctl enable ceph-volume@lvm-5-c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-5-c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90.service to /usr/lib/systemd/system/ceph-volume@.service. Running command: /bin/systemctl enable --runtime ceph-osd@5 stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)5.service to /usr/lib/systemd/system/ceph-osd@.service. Running command: /bin/systemctl start ceph-osd@5 stderr: Job for ceph-osd(a)5.service failed because the control process exited with error code. See "systemctl status ceph-osd(a)5.service" and "journalctl -xe" for details. --> Was unable to complete a new OSD, will rollback changes Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring osd purge-new osd.5 --yes-i-really-mean-it stderr: purged osd.5 --> RuntimeError: command returned non-zero exit status: 1 So I break the procedure into steps, and proceed them one by one. I saw the problem happen with the bluestore. [xxx@xxx.com@xxx430 ~]$ sudo ceph-osd -c /etc/ceph/${CLUSTER_NAME}.conf -k /etc/ceph/${CLUSTER_NAME}.client.admin.keyring -i $ID --mkfs \ > --monmap /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/activate.monmap --osd-uuid $UUID --no-mon-config \ > --osd-data /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/ --setuser ceph --setgroup ceph 2021-01-21 16:59:14.907 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5//block) _read_bdev_label failed to open /var/lib/ceph/osd/aap-storage-5//block: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5//block) _read_bdev_label failed to open /var/lib/ceph/osd/aap-storage-5//block: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _setup_block_symlink_or_file failed to open block file: (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) mkfs failed, (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied 2021-01-21 16:59:14.908 7f724e882a80 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/aap-storage-5/: (13) Permission denied Because of the permission problem, so I avoid using the ceph user for the ceph service, however, the problem still persisted. [xxx@xxx.com@xxx430 ~]$ sudo ceph-osd -c /etc/ceph/${CLUSTER_NAME}.conf -k /etc/ceph/${CLUSTER_NAME}.client.admin.keyring -i $ID --mkfs --monmap /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/activate.monmap --osd-uuid $UUID --no-mon-config --osd-data /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/ 2021-01-21 17:00:08.583 7f3acc546a80 -1 bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid -- KPMG IT Service GmbH Sitz/Registergericht: Berlin/Amtsgericht Charlottenburg, HRB 87521 B Geschäftsführer: Hans-Christian Schwieger, Helmar Symmank Aufsichtsratsvorsitzender: WP StB Klaus Becker Allgemeine Informationen zur Datenverarbeitung im Rahmen unserer allgemeinen Geschäftstätigkeit sowie im Mandatsverhältnis gemäß EU Datenschutz-Grundverordnung sind hier <https://home.kpmg.com/content/dam/kpmg/de/pdf/Themen/2018/datenschutzinform…> abrufbar. Die Information in dieser E-Mail ist vertraulich und kann dem Berufsgeheimnis unterliegen. Sie ist ausschließlich für den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, ist Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Maßnahmen im Vertrauen auf erlangte Information untersagt. In dieser E-Mail enthaltene Meinungen oder Empfehlungen unterliegen den Bedingungen des jeweiligen Mandatsverhältnisses mit dem Adressaten. The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter.

1 0

mds openfiles table shards
by Dan van der Ster 22 Jan '21

22 Jan '21

Hi all, During rejoin an MDS can sometimes go OOM if the openfiles table is too large. The workaround has been described by ceph devs as "rados rm -p cephfs_metadata mds0_openfiles.0". On our cluster we have several such objects for rank 0: mds0_openfiles.0 exists with size: 199978 mds0_openfiles.1 exists with size: 153650 mds0_openfiles.2 exists with size: 40987 mds0_openfiles.3 exists with size: 7746 mds0_openfiles.4 exists with size: 413 If we suffer such an OOM, do we need to rm *all* of those objects or only the `.0` object? Best Regards, Dan

1 1

Large rbd
by Chris Dunlop 22 Jan '21

22 Jan '21

Hi, What limits are there on the "reasonable size" of an rbd? E.g. when I try to create a 1 PB rbd with default 4 MiB objects on my octopus cluster: $ rbd create --size 1P --data-pool rbd.ec rbd.meta/fs 2021-01-20T18:19:35.799+1100 7f47a99253c0 -1 librbd::image::CreateRequest: validate_layout: image size not compatible with object ...which somes from: == src/librbd/image/CreateRequest.cc bool validate_layout(CephContext *cct, uint64_t size, file_layout_t &layout) { if (!librbd::ObjectMap<>::is_compatible(layout, size)) { lderr(cct) << "image size not compatible with object map" << dendl; return false; } == src/librbd/ObjectMap.cc template <typename I> bool ObjectMap<I>::is_compatible(const file_layout_t& layout, uint64_t size) { uint64_t object_count = Striper::get_num_objects(layout, size); return (object_count <= cls::rbd::MAX_OBJECT_MAP_OBJECT_COUNT); } == src/cls/rbd/cls_rbd_types.h static const uint32_t MAX_OBJECT_MAP_OBJECT_COUNT = 256000000; For 4 MiB objects that object count equates to just over 976 TB. Is there any particular reason for that MAX_OBJECT_MAP_OBJECT_COUNT, or it just "this is crazy large, if you're trying to go over this you're doing something wrong, rethink your life..."? Yes, I realise I can increase the size of the objects to get a larger rbd, or drop the object-map support (and the fast-diff that goes along with it). I'm SO glad I found this limit now rather than starting on a smaller rbd and a finding the limit when I tried to grow the rbd underneath a rapidly filling filesystem. What else should I know? Background: I currently have nearly 0.5 PB on XFS (on lvm / raid6) and ZFS that I'm looking to move over to ceph. XFS is a requirement, for the reflinking (sadly not yet available in CephFS: https://tracker.ceph.com/issues/1680) The recommendation for XFS is to start larger, on a thin-provisioned store (hello rbd!), rather than start smaller and grow as needed - e.g. see the thread surrounding: https://www.spinics.net/lists/linux-xfs/msg20099.html Rather than a single large rbd, should I be looking at multiple smaller rbds linked together using lvm or somesuch? What are the tradeoffs? And whilst we're here... for an rbd with the data on an erasure-coded pool, how do you calculate the amount of rbd metadata required if/when the rbd data is fully allocated? Cheers, Chris

8 9

RBD-Mirror Snapshot Scalability
by Adam Boyhan 21 Jan '21

21 Jan '21

I have noticed that RBD-Mirror snapshot mode can only manage to take 1 snapshot per second. For example I have 21 images in a single pool. When the schedule is triggered it takes the mirror snapshot of each image 1 at a time. It doesn't feel or look like a performance issue as the OSD's are Micron 9300 PRO NVMe's and each server has 2x Intel Platinum 8268 CPU's. I was hoping that adding more RDB-Mirror instance would help, but that only seems to help with overall throughput. As it sits I have 3 RBD-Mirror instances running on each cluster. We run a 30 minute snapshot schedule to our remote site as it is, based on that I can only squeeze 1800 mirror snaps every 30 minutes. I was hoping there might be something I am missing with RBD-Mirror as far as scaling goes. Maybe multiple pools would be a solution and have other benefits?

2 3

RBD-Mirror Mirror Snapshot stuck
by Adam Boyhan 21 Jan '21

21 Jan '21

I have a rbd-mirror snapshot on 1 image that failed to replicate and now its not getting cleaned up. The cause of this was my fault based on my steps. Just trying to understand how to clean up/handle the situation. Here is how I got into this situation. - Created manual rbd snapshot on the image - On the remote cluster I cloned the snapshot - While cloned on the secondary cluster I made the mistake of deleting the snapshot on the primary - The subsequent mirror snapshot failed - I then removed the clone - The next mirror snapshot was successful but I was left with this mirror snapshot on the primary that I can't seem to get rid of root@Ccscephtest1:/var/log/ceph# rbd snap ls --all CephTestPool1/vm-100-disk-0 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 10082 .mirror.primary.90c53c21-6951-4218-9f07-9e983d490993.e0c63479-b09e-4c66-a65b-085b67a19907 2 TiB Thu Jan 21 07:10:09 2021 mirror (primary peer_uuids:[]) 10243 .mirror.primary.90c53c21-6951-4218-9f07-9e983d490993.483e55aa-2f64-4bb0-ac0f-7b5aac59830e 2 TiB Thu Jan 21 07:30:08 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770]) I have tried deleting the snap with "rbd snap rm" like normal user created snaps, but no luck. Anyway to force the deletion?

2 2

How to make HEALTH_ERR quickly and pain-free
by George Shuklin 21 Jan '21

21 Jan '21

I have hell of the question: how to make HEALTH_ERR status for a cluster without consequences? I'm working on CI tests and I need to check if our reaction to HEALTH_ERR is good. For this I need to take an empty cluster with an empty pool and do something. Preferably quick and reversible. For HEALTH_WARN the best thing I found is to change pool size to 1, it raises "1 pool(s) have no replicas configured" warning almost instantly and it can be reverted very quickly for empty pool. But HEALTH_ERR is a bit more tricky. Any ideas?

2 5

cephfs: massive drop in MDS requests per second with increasing number of caps
by Dietmar Rieder 20 Jan '21

20 Jan '21

Hi all, we noticed a massive drop in requests per second a cephfs client is able to perform when we do a recursive chown over a directory with millions of files. As soon as we see about 170k caps on the MDS, the client performance drops from about 660 reqs/sec to 70 reqs/sec. When we then clear dentries and inodes using "sync; echo 2 > /proc/sys/vm/drop_caches" on the client, the request go up to ~660 again just to drop again when reaching about 170k caps. See the attached screenshots. When we stop the chown process for a while and restart it ~25min later again it still performs very slowly and the MDS reqs/sec remain low (~60/sec.). Clearing the cache (dentries and inodes) on the client restores the performance again. When we run the same chown on another client in parallel, it starts again with reasonable good performance (while the first client is poorly performing) but eventually it gets slow again just like the first client. Can someone comment on this and explain it? How can this be solved, so that the performance remains stable? We are running ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) on all ceph cluster nodes and on all clients. The OS on all ceph cluster nodes and client nodes is CentOS 7.9. The filesystem is mounted via CentOS kernel client (latest official version). Thanks in advance. ~Best Dietmar -- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Phone: +43 512 9003 71402 Fax: +43 512 9003 73100 Email: dietmar.rieder(a)i-med.ac.at Web: http://www.icbi.at

3 8

RBD on windows
by Szabo, Istvan (Agoda) 20 Jan '21

20 Jan '21

Hi, I'm looking the suse documentation regarding their option to have rbd on win. I want to try on windows server 2019 vm, but I got this error: PS C:\Users\$admin$> rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring C:/ProgramData/ceph/keyring 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load C:/ProgramData/ceph/keyring: (5) Input/output error 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load C:/ProgramData/ceph/keyring: (5) Input/output error 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input rbd: couldn't connect to the cluster! 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load C:/ProgramData/ceph/keyring: (5) Input/output error 2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 monclient: keyring not found This is the keyring file: [client.windowstest] key = AQBJ7wdgdWLIMhAAle+/pg+26XvWsDv8PyPcvw== caps mon = "allow rw" caps osd = "allow rwx pool=windowstest" And this is the ceph.conf file on the windows client: [global] log to stderr = true run dir = C:/ProgramData/ceph crash dir = C:/ProgramData/ceph [client] keyring = C:/ProgramData/ceph/keyring log file = C:/ProgramData/ceph/$name.$pid.log admin socket = C:/ProgramData/ceph/$name.$pid.asok [global] mon host = [v2:10.118.199.231:3300,v1:10.118.199.231:6789] [v2:10.118.199.232:3300,v1:10.118.199.232:6789] [v2:10.118.199.233:3300,v1:10.118.199.233:6789] Commands I've tried: rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring C:/ProgramData/ceph/keyring rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring C:\ProgramData\ceph\keyring rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring "C:/ProgramData/ceph/keyring" rbd create image01 --size 4096 --pool windowstest -m 10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring "C:\ProgramData\ceph\keyring" rbd create blank_image --size=1G The ceph version is luminous 12.2.8. Don't know why they don't find mon keyring. Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

[Query] Safe to discard bucket lock objects in reshard pool?
by Prasad Krishnan 20 Jan '21

20 Jan '21

Dear Ceph users, We have a slightly dated version of Luminous cluster in which dynamic bucket resharding was accidentally enabled due to a misconfig (we don't use this feature since the number of objects per bucket is capped). This resulted in creation of the RGW reshard pool with lots of bucket reshard lock objects (we have thousands of buckets) which is leading to clutter. Also, we've run into a malloc failure issue (similar to https://tracker.ceph.com/issues/21826 but not the same since we already use tcmalloc) on the OSDs in which these reshard lock objects are located and we'd like to reduce the objects that have to be copied out. My question to the community is: "Is it safe to discard the bucket reshard lock objects if we know that we'll never use the reshard feature on the cluster again?". The RGWs performed resharding several months ago due to a misconfiguration and we already have stale bucket instances which are due for cleanup on this cluster. Thanks, Prasad Krishnan -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

3 2

Re: Samsung PM883 3.84TB SSD performance
by mj 19 Jan '21

19 Jan '21

Hi both, Thanks for both quick replies, and both (of course) 100% spot-on! With 4k, IOPS is around 18530 :-) Thank you both, and apologies for the noise! Best, MJ On 19/01/2021 14:57, Marc Roos wrote: > You should test with 4k not 4M. > > >> -----Original Message----- >> From: mj <lists(a)merit.unu.edu> >> Sent: 19 January 2021 14:33 >> To: ceph-users <ceph-users(a)ceph.io> >> Subject: [ceph-users] Samsung PM883 3.84TB SSD performance >> >> Hi, >> >> We purchased Samsung PM883 3.84TB SSDs to be used as BlueStore SSDs in >> our cluster. >> >> I ran some benchmarks (write cache disabled, and with -sync=1 and >> -fsync=1) but with my little understandig, the results look terribly >> bad. (iops around 82!) >> >> Reading pages like yourcmc.ru, we should at least aim for > 1000s IOPS. >> (like 10000 up to 20000) >> >> Here is our complete output: >> >>> root@pve:~# hdparm -W 0 /dev/sdd >>> >>> /dev/sdd: >>> setting drive write-caching to 0 (off) >>> write-caching = 0 (off) >>> root@pve:~# fio -ioengine=libaio -fsync=1 -invalidate=1 -name=test - >> bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd >>> test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, >> iodepth=32 >>> fio-2.16 >>> Starting 1 process >>> Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta >> 00m:00s] >>> test: (groupid=0, jobs=1): err= 0: pid=3268778: Tue Jan 19 14:13:44 >> 2021 >>> write: io=19728MB, bw=336674KB/s, iops=82, runt= 60003msec >>> slat (usec): min=1853, max=4652, avg=2428.94, stdev=361.92 >>> clat (msec): min=9, max=652, avg=376.87, stdev=22.88 >>> lat (msec): min=11, max=655, avg=379.30, stdev=22.94 >>> clat percentiles (msec): >>> | 1.00th=[ 363], 5.00th=[ 367], 10.00th=[ 371], 20.00th=[ >> 371], >>> | 30.00th=[ 371], 40.00th=[ 375], 50.00th=[ 375], 60.00th=[ >> 379], >>> | 70.00th=[ 379], 80.00th=[ 383], 90.00th=[ 388], 95.00th=[ >> 392], >>> | 99.00th=[ 404], 99.50th=[ 420], 99.90th=[ 611], 99.95th=[ >> 635], >>> | 99.99th=[ 652] >>> lat (msec) : 10=0.02%, 50=0.06%, 100=0.10%, 250=0.24%, 500=99.25% >>> lat (msec) : 750=0.32% >>> cpu : usr=2.95%, sys=28.30%, ctx=47835, majf=0, minf=26 >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >>> =64=0.0% >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>> =64=0.0% >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >>> =64=0.0% >>> issued : total=r=0/w=4932/d=0, short=r=0/w=0/d=0, >> drop=r=0/w=0/d=0 >>> latency : target=0, window=0, percentile=100.00%, depth=32 >>> >>> Run status group 0 (all jobs): >>> WRITE: io=19728MB, aggrb=336674KB/s, minb=336674KB/s, >> maxb=336674KB/s, mint=60003msec, maxt=60003msec >>> >>> Disk stats (read/write): >>> sdd: ios=41/39330, merge=0/4995672, ticks=8/231816, in_queue=231440, >> util=79.43% >>> root@pve:~# fio -ioengine=libaio -sync=1 -invalidate=1 -name=test - >> bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd >>> test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, >> iodepth=32 >>> fio-2.16 >>> Starting 1 process >>> Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta >> 00m:00s] >>> test: (groupid=0, jobs=1): err= 0: pid=3270282: Tue Jan 19 14:14:58 >> 2021 >>> write: io=19468MB, bw=332248KB/s, iops=81, runt= 60001msec >>> slat (usec): min=11658, max=14287, avg=12318.59, stdev=371.39 >>> clat (usec): min=8, max=400078, avg=380931.63, stdev=19107.88 >>> lat (msec): min=11, max=412, avg=393.25, stdev=19.16 >>> clat percentiles (msec): >>> | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ >> 375], >>> | 30.00th=[ 375], 40.00th=[ 379], 50.00th=[ 379], 60.00th=[ >> 383], >>> | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 392], 95.00th=[ >> 396], >>> | 99.00th=[ 396], 99.50th=[ 400], 99.90th=[ 400], 99.95th=[ >> 400], >>> | 99.99th=[ 400] >>> lat (usec) : 10=0.02% >>> lat (msec) : 20=0.02%, 50=0.04%, 100=0.08%, 250=0.25%, 500=99.59% >>> cpu : usr=2.22%, sys=30.01%, ctx=53455, majf=0, minf=25 >>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >>> =64=0.0% >>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>> =64=0.0% >>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >>> =64=0.0% >>> issued : total=r=0/w=4867/d=0, short=r=0/w=0/d=0, >> drop=r=0/w=0/d=0 >>> latency : target=0, window=0, percentile=100.00%, depth=32 >>> >>> Run status group 0 (all jobs): >>> WRITE: io=19468MB, aggrb=332248KB/s, minb=332248KB/s, >> maxb=332248KB/s, mint=60001msec, maxt=60001msec >>> >>> Disk stats (read/write): >>> sdd: ios=41/38830, merge=0/4931664, ticks=4/203944, in_queue=203700, >> util=79.55% >>> >>> >> >> The disk seems to be properly detected properly by the OS: >> >>> [4145961.899584] scsi 0:0:16:0: Direct-Access ATA SAMSUNG >> MZ7LH3T8 904Q PQ: 0 ANSI: 6 >>> [4145961.899596] scsi 0:0:16:0: SATA: handle(0x001a), >> sas_addr(0x500304801ef80883), phy(3), device_name(0x0000000000000000) >>> [4145961.899598] scsi 0:0:16:0: enclosure logical id >> (0x500304801ef808bf), slot(3) >>> [4145961.899599] scsi 0:0:16:0: enclosure level(0x0000), connector >> name( ) >>> [4145961.899649] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), >> smart(y), fua(y), sw_preserve(y) >>> [4145961.900471] sd 0:0:16:0: Power-on or device reset occurred >>> [4145961.900544] sd 0:0:16:0: Attached scsi generic sg3 type 0 >>> [4145961.902367] sd 0:0:16:0: [sdd] 7501476528 512-byte logical >> blocks: (3.84 TB/3.49 TiB) >>> [4145961.902369] sd 0:0:16:0: [sdd] 4096-byte physical blocks >>> [4145961.904264] sd 0:0:16:0: [sdd] Write Protect is off >>> [4145961.904266] sd 0:0:16:0: [sdd] Mode Sense: 9b 00 10 08 >>> [4145961.904756] sd 0:0:16:0: [sdd] Write cache: enabled, read cache: >> enabled, supports DPO and FUA >>> [4145961.915996] sd 0:0:16:0: [sdd] Attached SCSI disk >> >> Anyone with a idea what we could be doing wrong? Or are these disks >> really unsuitable for OSD use? >> >> MJ >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Samsung PM883 3.84TB SSD performance
by mj 19 Jan '21

19 Jan '21

Hi, We purchased Samsung PM883 3.84TB SSDs to be used as BlueStore SSDs in our cluster. I ran some benchmarks (write cache disabled, and with -sync=1 and -fsync=1) but with my little understandig, the results look terribly bad. (iops around 82!) Reading pages like yourcmc.ru, we should at least aim for > 1000s IOPS. (like 10000 up to 20000) Here is our complete output: > root@pve:~# hdparm -W 0 /dev/sdd > > /dev/sdd: > setting drive write-caching to 0 (off) > write-caching = 0 (off) > root@pve:~# fio -ioengine=libaio -fsync=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd > test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=32 > fio-2.16 > Starting 1 process > Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=3268778: Tue Jan 19 14:13:44 2021 > write: io=19728MB, bw=336674KB/s, iops=82, runt= 60003msec > slat (usec): min=1853, max=4652, avg=2428.94, stdev=361.92 > clat (msec): min=9, max=652, avg=376.87, stdev=22.88 > lat (msec): min=11, max=655, avg=379.30, stdev=22.94 > clat percentiles (msec): > | 1.00th=[ 363], 5.00th=[ 367], 10.00th=[ 371], 20.00th=[ 371], > | 30.00th=[ 371], 40.00th=[ 375], 50.00th=[ 375], 60.00th=[ 379], > | 70.00th=[ 379], 80.00th=[ 383], 90.00th=[ 388], 95.00th=[ 392], > | 99.00th=[ 404], 99.50th=[ 420], 99.90th=[ 611], 99.95th=[ 635], > | 99.99th=[ 652] > lat (msec) : 10=0.02%, 50=0.06%, 100=0.10%, 250=0.24%, 500=99.25% > lat (msec) : 750=0.32% > cpu : usr=2.95%, sys=28.30%, ctx=47835, majf=0, minf=26 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% > issued : total=r=0/w=4932/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > WRITE: io=19728MB, aggrb=336674KB/s, minb=336674KB/s, maxb=336674KB/s, mint=60003msec, maxt=60003msec > > Disk stats (read/write): > sdd: ios=41/39330, merge=0/4995672, ticks=8/231816, in_queue=231440, util=79.43% > root@pve:~# fio -ioengine=libaio -sync=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd > test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=32 > fio-2.16 > Starting 1 process > Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=3270282: Tue Jan 19 14:14:58 2021 > write: io=19468MB, bw=332248KB/s, iops=81, runt= 60001msec > slat (usec): min=11658, max=14287, avg=12318.59, stdev=371.39 > clat (usec): min=8, max=400078, avg=380931.63, stdev=19107.88 > lat (msec): min=11, max=412, avg=393.25, stdev=19.16 > clat percentiles (msec): > | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 375], > | 30.00th=[ 375], 40.00th=[ 379], 50.00th=[ 379], 60.00th=[ 383], > | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 392], 95.00th=[ 396], > | 99.00th=[ 396], 99.50th=[ 400], 99.90th=[ 400], 99.95th=[ 400], > | 99.99th=[ 400] > lat (usec) : 10=0.02% > lat (msec) : 20=0.02%, 50=0.04%, 100=0.08%, 250=0.25%, 500=99.59% > cpu : usr=2.22%, sys=30.01%, ctx=53455, majf=0, minf=25 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% > issued : total=r=0/w=4867/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > WRITE: io=19468MB, aggrb=332248KB/s, minb=332248KB/s, maxb=332248KB/s, mint=60001msec, maxt=60001msec > > Disk stats (read/write): > sdd: ios=41/38830, merge=0/4931664, ticks=4/203944, in_queue=203700, util=79.55% > > The disk seems to be properly detected properly by the OS: > [4145961.899584] scsi 0:0:16:0: Direct-Access ATA SAMSUNG MZ7LH3T8 904Q PQ: 0 ANSI: 6 > [4145961.899596] scsi 0:0:16:0: SATA: handle(0x001a), sas_addr(0x500304801ef80883), phy(3), device_name(0x0000000000000000) > [4145961.899598] scsi 0:0:16:0: enclosure logical id (0x500304801ef808bf), slot(3) > [4145961.899599] scsi 0:0:16:0: enclosure level(0x0000), connector name( ) > [4145961.899649] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) > [4145961.900471] sd 0:0:16:0: Power-on or device reset occurred > [4145961.900544] sd 0:0:16:0: Attached scsi generic sg3 type 0 > [4145961.902367] sd 0:0:16:0: [sdd] 7501476528 512-byte logical blocks: (3.84 TB/3.49 TiB) > [4145961.902369] sd 0:0:16:0: [sdd] 4096-byte physical blocks > [4145961.904264] sd 0:0:16:0: [sdd] Write Protect is off > [4145961.904266] sd 0:0:16:0: [sdd] Mode Sense: 9b 00 10 08 > [4145961.904756] sd 0:0:16:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA > [4145961.915996] sd 0:0:16:0: [sdd] Attached SCSI disk Anyone with a idea what we could be doing wrong? Or are these disks really unsuitable for OSD use? MJ

2 1

Dashboard : Block image listing and infos
by Gilles Mocellin 18 Jan '21

18 Jan '21

Hello Cephers, On a new cluster, I only have 2 RBD block images, and the Dashboard doesn't manage to list them correctly. I have this message : Warning Displaying previously cached data for pool veeam-repos. Sometime it disappears, but as soon as I reload or return to the listing page, it's there. What I've seen, is a high CPU load due to ceph-mgr on the active manager. And also stack-traces like this : 2021-01-15T14:41:12.061+0100 7f7f3fec4700 0 [dashboard ERROR exception] Dashboard Exception Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 94, in dashboard_exception_handler return handler(*args, **kwargs) File "/usr/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 60, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 666, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 861, in wrapper return func(*vpath, **params) File "/usr/lib/python3.6/contextlib.py", line 52, in inner return func(*args, **kwds) File "/usr/lib/python3.6/contextlib.py", line 52, in inner return func(*args, **kwds) File "/usr/share/ceph/mgr/dashboard/controllers/rbd.py", line 86, in list return self._rbd_list(pool_name) File "/usr/share/ceph/mgr/dashboard/controllers/rbd.py", line 76, in _rbd_list status, value = RbdService.rbd_pool_list(pool) File "/usr/share/ceph/mgr/dashboard/tools.py", line 254, in wrapper return rvc.run(fn, args, kwargs) File "/usr/share/ceph/mgr/dashboard/tools.py", line 242, in run raise ViewCacheNoDataException() dashboard.exceptions.ViewCacheNoDataException: ViewCache: unable to retrieve data Also that, since I changed some features back and forth on one image : 2021-01-18T11:13:26.383+0100 7f00199ca700 0 [dashboard ERROR frontend.error] (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam- repos%252Fveeam-repo2-vol1): Cannot read property 'features_name' of undefined TypeError: Cannot read property 'features_name' of undefined at https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/1.9e79c41bbaed982a50af.js:1:121… at Array.forEach (<anonymous>) at R.deepBoxCheck (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/1.9e79c41bbaed982a50af.js:1:120…) at R.featureFormUpdate (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/1.9e79c41bbaed982a50af.js:1:121…) at https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/1.9e79c41bbaed982a50af.js:1:119… at d.a [as _next] (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) at d.__tryOrUnsub (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) at d.next (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) at l._next (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) at l.next (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) But that's perhaps just becaus I open an Edit window on the image and it does not have the datas. The Edit window is empty, and I can't edit things, especially, I wan't to resize the image. Finally, I find a similar bug already registered here, but that seems resolved for the owner : https://tracker.ceph.com/issues/45308 Mmmm, as I read carrefully, it'about cephfs not rbd in that stacktrace... Am I the only one ? Is there a work around ? I really need the dashboard to be usable, because I want to delegate a maximum of operations to people that don't need to have the rights to connect to machines to use CLI. And also don't have the skills to do so. Have a good day, -- Gilles

1 0

RBD image size in prometheus
by Seena Fallah 18 Jan '21

18 Jan '21

Hi, Is there any reason why RBD image size isn't exported in the Prometheus module? Thanks.

1 0

PG inconsistent with empty inconsistent objects
by Seena Fallah 17 Jan '21

17 Jan '21

Hi, I'm facing something strange! One of the PGs in my pool got inconsistent and when I run `rados list-inconsistent-obj $PG_ID --format=json-pretty` the `inconsistents` key was empty! What is this? Is it a bug in Ceph or..? Thanks.

3 4

Ceph 15.2.3 on Ubuntu 20.04 with odroid xu4 / python thread Problem
by Dominik H 16 Jan '21

16 Jan '21

Hi all, i try to run ceph client tools on an odroid xu4 (armhf) with Ubuntu 20.04 on python 3.8.5. Unfortunately there is the following error on each "ceph" command (even in ceph --help) Traceback (most recent call last): File "/usr/bin/ceph", line 1275, in <module> retval = main() File "/usr/bin/ceph", line 981, in main cluster_handle = run_in_thread(rados.Rados, File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1342, in run_in_thread raise Exception("timed out") Exception: timed out With this Server I access an existing Ceph-Cluster with the same hardware. I checked the code part, there is just a thread start and a join (waiting for finish a RadosThread). Maybe this is a python error in combination with armhf architecture? Maybe someone can help. Thanks and greetings Dominik

3 8

librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.
by Rafael Diaz Maurin 15 Jan '21

15 Jan '21

Hello cephers, I run Nautilus (14.2.15) Here is my context : each night a script take a snapshot from each RBD volume in a pool (all the disks of the VMs hosted) on my ceph production cluster. Then each snapshot is exported (rbd export-diff | rbd import-diff in SSH) towards my ceph bakup cluster. 0. Yesterday, and for the firt time, I had this error when I ran ''rbd -p MY_POOL du'' on the ceph backup cluster : 021-01-14 11:41:29.294 7f1f2fbcf0c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map *rbd_object_map.781776950f8dd5.000000000000b740* 1. I tried to find the image concerned by the error, and I find this one, with a slight diffence with the number of the object-map : rbd image 'MY_IMAGE': size 30 GiB in 7680 objects order 22 (4 MiB objects) snapshot_count: 32 id: 781776950f8dd5 block_name_prefix: *rbd_data.781776950f8dd5* format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, operations op_features: snap-trash flags: create_timestamp: Thu Mar 19 01:00:27 2020 access_timestamp: Thu Mar 19 01:00:27 2020 modify_timestamp: Thu Jan 14 01:42:23 2021 2. Then I ran a check wich not show any error rbd -p MY_POOL object-map check "MY_IMAGE" Object Map Check: 100% complete...done. 3. Then, I still ran a rebuild rbd -p MY_POOL object-map rebuild "MY_IMAGE" Object Map Rebuild: 100% complete...done. 4. But the error is still here : 2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740 Do someone know how to find the the image concerned by the object-map error and fix this error. Thank you for your help, Rafael

3 5

Python API mon_comand()
by Robert Sander 15 Jan '21

15 Jan '21

Hi, I am trying to get some statistics via the Python API but fail to run the equivalent of "ceph df detail". One the command line I get: # ceph -f json df |jq .pools[0] { "name": "rbd", "id": 1, "stats": { "stored": 27410520278, "objects": 6781, "kb_used": 80382849, "bytes_used": 82312036566, "percent_used": 0.1416085809469223, "max_avail": 166317473792 } } # ceph -f json df detail |jq .pools[0] { "name": "rbd", "id": 1, "stats": { "stored": 27410520278, "objects": 6781, "kb_used": 80382849, "bytes_used": 82312036566, "percent_used": 0.1416085809469223, "max_avail": 166317473792, "quota_objects": 0, "quota_bytes": 0, "dirty": 6781, "rd": 309130743, "rd_bytes": 327278814208, "wr": 155492443, "wr_bytes": 155528225792, "compress_bytes_used": 0, "compress_under_bytes": 0, "stored_raw": 82231558358, "avail_raw": 498952444191 } } In Python I just get an error: # python Python 2.7.16 (default, Oct 10 2019, 22:02:15) [GCC 8.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import json >>> import rados >>> cluster = rados.Rados(conffile='/etc/ceph/ceph.conf') >>> cluster.connect() >>> cluster.mon_command(json.dumps({'prefix': 'df', 'format': 'json'}), b'') (0, '{"stats":{"total_bytes":643914731520,"total_avail_bytes":550794231808,"total_used_bytes":86678048768,"total_used_raw_bytes":93120499712,"total_used_raw_ratio":0.14461618661880493,"num_osds":6,"num_per_pool_osds":6},"stats_by_class":{"hdd":{"total_bytes":643914731520,"total_avail_bytes":550794231808,"total_used_bytes":86678048768,"total_used_raw_bytes":93120499712,"total_used_raw_ratio":0.14461618661880493}},"pools":[{"name":"rbd","id":1,"stats":{"stored":27410520278,"objects":6781,"kb_used":80382849,"bytes_used":82312036566,"percent_used":0.1416085809469223,"max_avail":166317473792}},{"name":"cephfs_data","id":3,"stats":{"stored":1282414464,"objects":307,"kb_used":3757248,"bytes_used":3847421952,"percent_used":0.0076519949361681938,"max_avail":166317473792}},{"name":"cephfs_metadata","id":4,"stats":{"stored":458803,"objects":22,"kb_used":2693,"bytes_used":2757248,"percent_used":5.5260434237425216e-06,"max_avail":166317473792}}]}\n', u'') >>> cluster.mon_command(json.dumps({'prefix': 'df detail', 'format': 'json'}), b'') (-22, '', u'command not known') >>> Anything I can do to get the output of "ceph df detail" via Python API? I would like to have the stats fields "rd", "wr", "rd_bytes" and "wr_bytes" per pool. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

.rgw.root was created wit a lot of PG
by Szabo, Istvan (Agoda) 15 Jan '21

15 Jan '21

Hi, Originally this pool was created with 512PG which makes couple of OSDs having 500PG 😲 What is the safe steps to copy over this pool? These are the files in this pool: default.realm period_config.f320e60d-8cff-4824-878e-c316423cc519 periods.18d63a25-8a50-4e17-9561-d452621f62fa.latest_epoch default.zonegroup.f320e60d-8cff-4824-878e-c316423cc519 zone_info.ba16656f-2191-40bb-bc39-9f19448d215d periods.6605bb4c-2226-4509-a3be-d5c95300fe14.1 default.zone.f320e60d-8cff-4824-878e-c316423cc519 zonegroup_info.f47a81ba-b214-4b8d-9b0e-84c14bc153cf periods.f320e60d-8cff-4824-878e-c316423cc519:staging realms.f320e60d-8cff-4824-878e-c316423cc519.control periods.6605bb4c-2226-4509-a3be-d5c95300fe14.latest_epoch realms.f320e60d-8cff-4824-878e-c316423cc519 realms_names.default periods.18d63a25-8a50-4e17-9561-d452621f62fa.1 zone_names.default zonegroups_names.default It’s single site cluster. Do I need to stop rados gateway? After I would follow the following steps: ceph osd pool create .rgw.root.new 8 rados cppool .rgw.root .rgw.root.new ceph osd pool delete .rgw.root .rgw.root --yes-i-really-really-mean-it ceph osd pool rename .rgw.root.new .rgw.root ceph osd pool application enable .rgw.root rgw Just not sure do I need to create outage request or not due to the rados stop. Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Centos 8 2021 with ceph, how to move forward?
by Szabo, Istvan (Agoda) 15 Jan '21

15 Jan '21

Hi, Just curious how you guys move forward with this Centos 8 change. We just finished installing our full multisite cluster and looks like we need to change the operating system. So curious if you are using centos 8 with ceph, where you are going to move forward. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

6 6

radosgw-admin sync status takes ages to print output
by Szabo, Istvan (Agoda) 15 Jan '21

15 Jan '21

Hello, I have a 3 DC octopus Multisite setup with bucket sync policy applied. I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 because they want to use 1 bucket but with a huge amount of objects (2.400.000.000 and 900.000.000) and in case of multisite we need to preshard the buckets as it is in the documentation. Do I need to fine tune something on the syncing to make this query faster? This is the output after 5-10 minutes query time not sure is it healthy or good or not to be honest, haven’t really find any good explanation about the output in the ceph documentation. From the master zone I can’r reallt even query because timed out, but in secondary zone can see this: radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:01:00.131104+0700 [11] source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:05:26.879014+0700 [98] Hope I can find some expert in the multisite area 😊 Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

How to reset an OSD
by Pfannes, Fabian 14 Jan '21

14 Jan '21

We running a small Ceph cluster with two nodes. Our failureDomain is set to host to have the data replicated between the two hosts. The other night one host crashed hard and three OSDs won't recovert with either debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 -1 osd.23 0 OSD::init() : unable to read osd superblock debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bluestore(/var/lib/ceph/osd/ceph-23) umount debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) allocation stats probe 0: cnt: 0 frags: 0 size: 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -1: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -2: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -4: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -8: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -16: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) ------------ debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 4 rocksdb: [db/db_impl.cc:563] Shutdown complete debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bluefs umount debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bdev(0x557150e20700 /var/lib/ceph/osd/ceph-23/block) close debug 2021-01-13T08:13:18.167+0000 7f9bfbd6ef40 1 freelist shutdown debug 2021-01-13T08:13:18.167+0000 7f9bfbd6ef40 1 bdev(0x557150e20000 /var/lib/ceph/osd/ceph-23/block) close debug 2021-01-13T08:13:18.411+0000 7f9bfbd6ef40 -1 ** ERROR: osd init failed: (22) Invalid argument or debug -2> 2021-01-13T08:13:29.991+0000 7f402c5f9700 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: expected 2795871023, got 2381104739 in db/000060.sst offset 748408 size 3819 code = 2 Rocksdb transaction: How can I delete the OSDs to get them back fully operational? Any help appreciated! /Fabian

2 2

Decoding pgmap
by George Shuklin 14 Jan '21

14 Jan '21

There is a command `ceph pg getmap`. It produces a binary file. Are there any utility to decode it?

1 0

Latency spike investigations on all SSD hardware cluster
by Martin Hronek 14 Jan '21

14 Jan '21

Hello fellow CEPH-users, we are currently investigating latency spikes in our CEPH(14.2.11) prod cluster, usually occurring when under heavy load. TLDR: Do you have an idea where to investigate some kv commit latency spikes on a CEPH cluster with a LSI 9300-8i HBA and all SSD(Intel, Micron) OSDs? The cluster consists of 3MDS nodes(2 active + 1 standby-replay), 3MON nodes(each running MGR+MON daemon) and 4OSD nodes(each having 8 SSD bluestore OSD disks). All nodes are running ubuntu latest 18.04 with kernel version 5.4(2 OSD Server still have 4.15 - but spike are seen on all of the OSD servers). As the spikes seem to be randomly distributed across time(under load) and OSD, we followed the spikes to find following messages on the OSD nodes: ``` bluestore(/var/lib/ceph/osd/ceph-31) log_latency slow operation observed for kv_sync, latency = 5.22298s bluestore(/var/lib/ceph/osd/ceph-31) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.5732s, txc = 0x55b1a98d9e00 ... bluestore(/var/lib/ceph/osd/ceph-31) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.50842s, txc = 0x55b1aa197800 bluestore(/var/lib/ceph/osd/ceph-31) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.5058s, txc = 0x55b1b7e75c00 ``` We found timely correlated kernel messages, which suggest that it might has something to do with the underlying SSDs ``` kernel: [3613612.312027] sd 4:0:10:0: attempting task abort!scmd(0x00000000dac86408), outstanding for 31384 ms & timeout 30000 ms kernel: [3613612.312034] sd 4:0:10:0: [sdg] tag#744 CDB: Write(10) 2a 00 be 11 b8 80 00 00 08 00 kernel: [3613612.312036] scsi target4:0:10: handle(0x0013), sas_address(0x4433221104000000), phy(4) kernel: [3613612.312038] scsi target4:0:10: enclosure logical id(0x500605b00e70a7b0), slot(7) kernel: [3613612.312039] scsi target4:0:10: enclosure level(0x0000), connector name( ) kernel: [3613612.312040] sd 4:0:10:0: No reference found at driver, assuming scmd(0x00000000dac86408) might have completed kernel: [3613612.312042] sd 4:0:10:0: task abort: SUCCESS scmd(0x00000000dac86408) ``` There are lots of the above blocks until the Kernel apparently has enough of them and just resets the device/interface: ``` kernel: [3613612.312267] sd 4:0:10:0: attempting task abort!scmd(0x00000000d7aaff5a), outstanding for 31388 ms & timeout 30000 ms kernel: [3613612.312269] sd 4:0:10:0: [sdg] tag#520 CDB: Write(10) 2a 00 be 11 b4 e0 00 00 08 00 kernel: [3613612.312269] scsi target4:0:10: handle(0x0013), sas_address(0x4433221104000000), phy(4) kernel: [3613612.312270] scsi target4:0:10: enclosure logical id(0x500605b00e70a7b0), slot(7) kernel: [3613612.312271] scsi target4:0:10: enclosure level(0x0000), connector name( ) kernel: [3613612.312272] sd 4:0:10:0: No reference found at driver, assuming scmd(0x00000000d7aaff5a) might have completed kernel: [3613612.312273] sd 4:0:10:0: task abort: SUCCESS scmd(0x00000000d7aaff5a) kernel: [3613612.653004] sd 4:0:10:0: Power-on or device reset occurred kernel: [3613613.254064] sd 4:0:10:0: Power-on or device reset occurred ``` OSD nodes are equipped with an "LSI 9300-8i SAS HBA" also we use two types of SSDs "Intel SSD D3-S4510 Series 1,92 TB", "Micron 5210 ION 1.92TB SSD". Since these resets happen on both SSDs we figured the least common denominator is the HBA so we did an Upgrade to the latest FW/BIOS on one OSD node. Sadly this did not solve the issue. * The question is now has someone a similar hardware configuration and issues with it ? * Do you have an idea what could be the cause of this behaviour ? * Or which part to investigate further ? Thanks for your hints and time reading :) M

1 0

Ceph RadosGW & OpenStack swift problem
by Mika Saari 14 Jan '21

14 Jan '21

Hi, Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW to work. I have managed to get the RadosGW working. I can manage it through a dashboard and use aws s3 client to create new buckets etc. When trying to use swift I get errors. Not sure how to continue to track the problem here. Any tips are welcome. Thank you very much, -Mika ------- What I have done and what are the results. Some data changed manually ------- What I have done: At OpenStack Side: 1) openstack user create --domain default --password-prompt swift 2) openstack role add --project service --user swift admin 3) openstack endpoint create --region RegionOne object-store public http://ceph1/swift/v1/AUTH_%$project_id$s 4) openstack endpoint create --region RegionOne object-store internal http://ceph1/swift/v1/AUTH_%$project_id$s 5) openstack endpoint create --region RegionOne object-store admin http://ceph1/swift/v1 At Ceph side: 1) ceph config set mgr rgw_keystone_api_version 3 2) ceph config set mgr rgw_keystone_url http://controller:5000 3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://controller:5000/v3/auth/tokens DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80 "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i http://ceph1/swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: http://ceph1/swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 401 Unauthorized b'AccessDenied'

4 12

OSDs in pool full : can't restart to clean
by Paul Mezzanini 14 Jan '21

14 Jan '21

Hey all We landed in a bad place (tm) with our nvme metadata tier. I'll root cause how we got here after it's all back up. I suspect it was a pool got misconfigured and just filled it all up. Short version, the OSDs are all full (or full enough) that I can't get them to spin back up. They crash with enospc. Average fragmentation for block is in the .8 range and bluefs-db is slightly better (using ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-412 free-score). I've tried all sorts of things. I was able to get a few to spin up but once they came up and rejoined they tried to pull MORE data in and crashed out again. I changed the crush_rule for the pool I care about to a much larger (and slower) set of disks. That way if I get anything else to come up I'm not just making it worse. I increased the size of the backing LV for one of the OSDs to see if I could get ceph-bluestore-tool to expand it, but that too crashes out enospc. In theory, there are a few pools I don't care about as much on there and I could delete them to make space, but I can't get them up enough -or- get the offline tools to do it. Some logs from the attempted expansion that fails: [root@ceph-b-07 ceph-412]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-412 bluefs-bdev-expand inferring bluefs devices from bluestore path 1 : device size 0x44aa000000 : own 0x[520000~20000,23e0000~620000,2ae0000~4d20000,78d0000~f30000,8900000~1600000,9fc0000~30000,a000000~5d00000,fe00000~3b00000,139e0000~5420000,19000000~100000, ::snip:: 4f0000~20000,25c17c0000~10000,25c2ea0000~20000,25c9f20000~10000,25d0860000~10000,25d50e0000~20000,25d5170000~10000,25ded20000~20000,25f4fc0000~20000] = 0x59c5b0000 : using 0x58f220000(22 GiB) : bluestore has 0x10260000(258 MiB) available Expanding DB/WAL... Expanding Main... 2021-01-13 16:40:46.481 7f33d1998ec0 -1 bluestore(/var/lib/ceph/osd/ceph-412) allocate_bluefs_freespace failed to allocate on 0x32c70000 min_size 0xf700000 > allocated total 0x1e80000 bluefs_shared_alloc_size 0x10000 allocated 0x1e80000 available 0x 90210000 2021-01-13 16:40:46.482 7f33d1998ec0 -1 bluefs _allocate failed to expand slow device to fit +0xf6f0def 2021-01-13 16:40:46.482 7f33d1998ec0 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0xf6f0def /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.15/rpm/el7/BUILD/ceph-14.2.15/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f33d1998ec0 time 2021-01-13 16:40:46.482978 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.15/rpm/el7/BUILD/ceph-14.2.15/src/os/bluestore/BlueFS.cc: 2351: ceph_abort_msg("bluefs enospc") ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) The orignal LV under that is 172g and the new LV size is double that. I'm going to keep poking at this, but I'm really hoping for some new info. Either to increase the size of the OSDs to get it back up enough so I can then rebuild them with a different layout, delete some data I don't care about, pull the data off and put it back to defrag... I don't care which so long as I get it back up. Thanks -paul

1 1

Which version of Ceph fully supports CephFS Snapshot?
by fantastic2085 13 Jan '21

13 Jan '21

I would like to use the Cephfs Snapshot feature, which version of Ceph supports it

3 2

Question about osdmap
by Andrea Bolzonella 12 Jan '21

12 Jan '21

Hi all. I'm running version 14.2.16. My cluster is remapping and backfilling after "reweight-by-utilization". This morning I started to have some issues: - all my mgr active and standby crashed - MDS stopped reporting statistics - MON is using lot of memory - Cluster was very slow I restarted everything but osd I managed to get the statistics back, restarting the MDS a couple of times. All above issues are gone but I have this "mon.node1 (mon.0) 1529395 : cluster [DBG] osdmap e??????:" about 1 every 2 second with a different epoc. Question: Is it normal or there is something wrong? Should be the osdmap more "stable" ? What can change the osdmap? Thanks for the help Andrea

1 0

Re: Global AVAIL vs Pool MAX AVAIL
by Mark Johnson 12 Jan '21

12 Jan '21

Thanks Anthony, Shortly after I made that post, I found a Server Fault post where someone had asked the exact same question. The reply was this - "The 'MAX AVAIL' column represents the amount of data that can be used before the first OSD becomes full. It takes into account the projected distribution of data across disks from the CRUSH map and uses the 'first OSD to fill up' as the target." To answer your question, yes we have a rather unbalanced cluster which is something I'm working on. When I saw these figures, I got scared that I had less time to work on it than I thought. There are about 10 pools in the cluster, but we primarily use one for almost all of our storage and it only has 64 pgs & 1 replica across 20 OSDs. So, as data has grown, it works out that each PG in this cluster accounts for about 148GB, and the OSDs are about 1.4TB each, so it's easy to see how it's found itself way out of balance. Anyway, once I've added the OSDs and data has rebalanced, I'm going to start the process of incrementally increasing the PG count for this pool in a staged process to reduce the amount of data per PG and (hopefully) balance out the data distribution better than it is. This is one big learning process - I just wish I wasn't learning in production so much. On Mon, 2021-01-11 at 15:58 -0800, Anthony D'Atri wrote: Either you have multiple CRUSH roots or device classes, or you have unbalanced OSD utilization. What version of Ceph? Do you have any balancing enabled? Do ceph osd df | sort -nk8 | head ceph osd df | sort -nk8 | tail and I’ll bet you have OSDs way more full than others. The STDDEV value that ceph df reports I suspect is accordingly high On Jan 11, 2021, at 2:07 PM, Mark Johnson < <mailto:markj@iovox.com> markj(a)iovox.com > wrote: Can someone please explain to me the difference between the Global "AVAIL" and the "MAX AVAIL" in the pools table when I do a "ceph df detail"? The reason being that we have a total of 14 pools, however almost all of our data exists in one pool. A "ceph df detail" shows the following: GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 28219G 6840G 19945G 70.68 36112k But the POOLS table from the same output shows the MAX AVAIL for each pool as 498G and the pool with all the data shows 9472G used with a %USED of 95.00. If it matters, the pool size is set to 2 so my guess is the global available figure is raw, meaning I should still have approx. 3.4TB available, but that 95% used has me concerned. I'm going to be adding some OSDs soon but still would like to understand the difference and how much trouble I'm in at this point. _______________________________________________ ceph-users mailing list -- <mailto:ceph-users@ceph.io> ceph-users(a)ceph.io To unsubscribe send an email to <mailto:ceph-users-leave@ceph.io> ceph-users-leave(a)ceph.io

2 1

Unable to cancel buckets from resharding queue
by Wout van Heeswijk 12 Jan '21

12 Jan '21

We are experiencing some issues with the bucket resharding queue in Ceph Mimic at one of our customers. I suspect that some of the issues are related to the upgrades of earlier versions of the cluster/radosgw. 1) When we cancel the resharding of a bucket, the bucket resharding entry is removed from the queue and almost immediately re-added. We confirm the removal by listing the omapkeys of all the reshard.0000## objects. The relevant omap key is temporarily removed. After a short time it is re-added. I haven't yet determined which process adds it back, but I can only think it is one of the two rgws. 2) We see a lot of objects,1265, in the reshard_pool: "default.rgw.log:reshard". Most look like they might be old to buckets markers or something like that. Most of these objects have not been touched in a long time (mtime 2018). Some numbers: Total buckets: 569 Objects in index pool: 649 Objects in default.rgw.log:reshard namespace: 1265 (of which 16 are the ‘rgw_reshard_num_logs’ objects) these are size '0' Buckets in reshard queue: 41 The objects in "default.rgw.log:reshard" have a similar naming schema to the index objects or , but I cannot relate them directly. Obviously there are loads more than the bucket index objects. The pools used for RGW are for an older generation of Radosgw: .rgw.root default.rgw.control default.rgw.data.root default.rgw.gc default.rgw.log default.rgw.users.uid default.rgw.users.keys default.rgw.buckets.index default.rgw.buckets.data default.rgw.users.email default.rgw.buckets.non-ec default.rgw.users.swift default.rgw.usage My two main questions are: 1) What process, other than dynamic resharding, could cause the re-adding of these buckets to the resharding queue? 2) Do more people see lots of objects in the reshard pool/namespace and can somebody help me understand what these object are? If somebody can point me in the direction of some more documentation or a good talk regarding the resharding mechanism that would also be great. Thanks and with kind regards, Wout 42on

1 0

Global AVAIL vs Pool MAX AVAIL
by Mark Johnson 11 Jan '21

11 Jan '21

Can someone please explain to me the difference between the Global "AVAIL" and the "MAX AVAIL" in the pools table when I do a "ceph df detail"? The reason being that we have a total of 14 pools, however almost all of our data exists in one pool. A "ceph df detail" shows the following: GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 28219G 6840G 19945G 70.68 36112k But the POOLS table from the same output shows the MAX AVAIL for each pool as 498G and the pool with all the data shows 9472G used with a %USED of 95.00. If it matters, the pool size is set to 2 so my guess is the global available figure is raw, meaning I should still have approx. 3.4TB available, but that 95% used has me concerned. I'm going to be adding some OSDs soon but still would like to understand the difference and how much trouble I'm in at this point.

1 0

bluefs_buffered_io=false performance regression
by Robert Sander 11 Jan '21

11 Jan '21

Hi, bluefs_buffered_io was disabled in Ceph version 14.2.11. The cluster started last year with 14.2.5 and got upgraded over the year now running 14.2.16. The performance was OK first but got abysmal bad at the end of 2020. We checked the components and HDDs and SSDs seem to be fine. Single disk benchmarks showed performance according the specs. Today we (re-)enabled bluefs_buffered_io and restarted all OSD processes on 248 HDDs distributed over 12 nodes. Now the benchmarks are fine again: 434MB/s write instead of 60MB/s, 960MB/s read instead of 123MB/s. This setting was disabled in 14.2.11 because "in some test cases it appears to cause excessive swap utilization by the linux kernel and a large negative performance impact after several hours of run time." We have to monitor if this will happen in our cluster. Is there any other negative side effect currently known? Here are the rados bench values, first with bluefs_buffered_io=false, then with bluefs_buffered_io=true: Bench Total Total Write Object Band Stddev Max Min Average Stddev Max Min Average Stddev Max Min time writes Read size width Bandwidth IOPS Latency (s) run reads size (MB/sec) made false write 33,081 490 4194304 4194304 59,2485 71,3829 264 0 14 17,8702 66 0 1,07362 2,83017 20,71 0,0741089 false seq 15,8226 490 4194304 4194304 123,874 30 46,8659 174 0 0,51453 9,53873 0,00343417 false rand 38,2615 2131 4194304 4194304 222,782 55 109,374 415 0 0,28191 12,1039 0,00327948 true write 30,4612 3308 4194304 4194304 434,389 26,0323 480 376 108 6,50809 120 94 0,14683 0,07368 0,99791 0,0751249 true seq 13,7628 3308 4194304 4194304 961,429 240 22,544 280 184 0,06528 0,88676 0,00338191 true rand 30,1007 8247 4194304 4194304 1095,92 273 25,5066 313 213 0,05719 0,99140 0,00325295 Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 3

denied reconnect attempt for ceph fs client
by Frank Schilder 11 Jan '21

11 Jan '21

Hi all, I'm not 100% sure, but I believe that since the update from mimic-13.2.8 to mimic-13.2.10 I have a strange issue. If a ceph fs client becomes unresponsive, it is evicted, but it cannot reconnect; see ceph.log extract below. In the past, clients would retry after the blacklist period and everything continued fine. I'm wondering why the clients cannot reconnect any more. I see this now every time a client gets thrown out and didn't have this problem before. Any hints to what I might want to change are welcome, and also information what might have changed during the update (eg. is this expected or not). 2021-01-11 19:45:53.839721 [INF] denied reconnect attempt (mds is up:active) from client.30770997 192.168.56.121:0/2325067585 after 1.82224e+06 (allowed interval 45) 2021-01-11 19:45:46.713822 [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2021-01-11 19:45:45.937126 [INF] MDS health message cleared (mds.0): 1 slow requests are blocked > 30 secs 2021-01-11 19:45:39.527180 [INF] Evicting (and blacklisting) client session 30770997 (192.168.56.121:0/2325067585) 2021-01-11 19:45:39.527168 [WRN] evicting unresponsive client HOSTNAME:CLIENT_NAME (30770997), after 62.735 seconds 2021-01-11 19:45:39.522141 [WRN] 1 slow requests, 0 included below; oldest blocked for > 50.371751 secs 2021-01-11 19:45:34.522085 [WRN] 1 slow requests, 0 included below; oldest blocked for > 45.371685 secs 2021-01-11 19:45:29.521991 [WRN] 1 slow requests, 0 included below; oldest blocked for > 40.371604 secs 2021-01-11 19:45:24.521907 [WRN] 1 slow requests, 0 included below; oldest blocked for > 35.371520 secs 2021-01-11 19:45:19.521895 [WRN] slow request 30.371469 seconds old, received at 2021-01-11 19:44:49.150361: client_request(client.30771333:10419033 getattr pAsLsXsFs #0x10014446b0f 2021-01-11 19:44:49.145012 caller_uid=257062, caller_gid=257062{}) currently failed to rdlock, waiting Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0