December 2020 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

Fwd: ceph-fuse false passed X_OK check
by Alex Taylor 30 Mar '21

30 Mar '21

Hi Patrick, Any updates? Looking forward to your reply :D On Thu, Dec 17, 2020 at 11:39 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote: > > On Wed, Dec 16, 2020 at 5:46 PM Alex Taylor <alexu4993(a)gmail.com> wrote: > > > > Hi Cephers, > > > > I'm using VSCode remote development with a docker server. It worked OK > > but fails to start the debugger after /root mounted by ceph-fuse. The > > log shows that the binary passes access X_OK check but cannot be > > actually executed. see: > > > > ``` > > strace_log: access("/root/.vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7", > > X_OK) = 0 > > > > root@develop:~# ls -alh > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > -rw-r--r-- 1 root root 978 Dec 10 13:06 > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > ``` > > > > I also test the access syscall on ext4, xfs and even cephfs kernel > > client, all of them return -EACCES, which is expected (the extension > > will then explicitly call chmod +x). > > > > After some digging in the code, I found it is probably caused by > > https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L5549-L5550. > > So here come two questions: > > 1. Is this a bug or is there any concern I missed? > > I tried reproducing it with the master branch and could not. It might > be due to an older fuse/ceph. I suggest you upgrade! > I tried the master(332a188d9b3c4eb5c5ad2720b7299913c5a772ee) as well and the issue still exists. My test program is: ``` #include <stdio.h> #include <unistd.h> int main() { int r; const char path[] = "test"; r = access(path, F_OK); printf("file exists: %d\n", r); r = access(path, X_OK); printf("file executable: %d\n", r); return 0; } ``` And the test result: ``` # local filesystem: ext4 root@f626800a6e85:~# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:13 test root@f626800a6e85:~# ./a.out file exists: 0 file executable: -1 root@f626800a6e85:~# findmnt -t fuse.ceph-fuse TARGET SOURCE FSTYPE OPTIONS /root/mnt ceph-fuse fuse.ceph-fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other root@f626800a6e85:~# cd mnt # ceph-fuse root@f626800a6e85:~/mnt# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:10 test root@f626800a6e85:~/mnt# ./a.out file exists: 0 file executable: 0 root@f626800a6e85:~/mnt# ./test bash: ./test: Permission denied ``` Again, ceph-fuse says file `test` is executable but in fact it can't be executed. The kernel version I'm testing on is: ``` root@f626800a6e85:~/mnt# uname -ar Linux f626800a6e85 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux ``` Please try the program above and make sure you're running it as root user, thank you. And if the reproduction still fails, please let me know the kernel version. > > 2. It works again with fuse_default_permissions=true, any drawbacks if > > this option is set? > > Correctness (ironically, for you) and performance. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >

2 1

CephFS max_file_size
by Mark Schouten 25 Mar '21

25 Mar '21

Hi, There is a default limit of 1TiB for the max_file_size in CephFS. I altered that to 2TiB, but I now got a request for storing a file up to 7TiB. I'd expect the limit to be there for a reason, but what is the risk of setting that value to say 10TiB? -- Mark Schouten <mark(a)tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208

5 4

fixing future rctimes
by Dan van der Ster 23 Mar '21

23 Mar '21

Hi all, We have a few subdirs with an rctime in the future. # getfattr -n ceph.dir.rctime session # file: session ceph.dir.rctime="2576387188.090" I can't find any subdir or item in that directory with that rctime, so I presume that there was previously a file and that rctime cannot go backwards [1] Is there any way to fix these rctimes so they show the latest ctime of the subtree? Also -- are we still relying on the client clock to set the rctime / ctime of a file? Would it make sense to limit ctime/rctime for any update to the current time on the MDS ? Best Regards, Dan [1] https://github.com/ceph/ceph/pull/24023/commits/920ef964311a61fcc6c0d6671b7…

2 3

Removing secondary data pool from mds
by Michael Thomas 13 Mar '21

13 Mar '21

I have a cephfs secondary (non-root) data pool with unfound and degraded objects that I have not been able to recover[1]. I created an additional data pool and used "setfattr -n ceph.dir.layout.pool' and a very long rsync to move the files off of the degraded pool and onto the new pool. This has completed, and using find + 'getfattr -n ceph.file.layout.pool', I verified that no files are using the old pool anymore. No ceph.dir.layout.pool attributes point to the old pool either. However, the old pool still reports that there are objects in the old pool, likely the same ones that were unfound/degraded from before: https://pastebin.com/qzVA7eZr Based on a old message from the mailing list[2], I checked the MDS for stray objects (ceph daemon mds.ceph4 dump cache file.txt ; grep -i stray file.txt) and found 36 stray entries in the cache: https://pastebin.com/MHkpw3DV. However, I'm not certain how to map these stray cache objects to clients that may be accessing them. 'rados -p fs.data.archive.frames ls' shows 145 objects. Looking at the parent of each object shows 2 strays: for obj in $(cat rados.ls.txt) ; do echo $obj ; rados -p fs.data.archive.frames getxattr $obj parent | strings ; done [...] 10000020fa1.00000000 10000020fa1 stray6 10000020fbc.00000000 10000020fbc stray6 [...] ...before getting stuck on one object for over 5 minutes (then I gave up): 1000005b1af.00000083 What can I do to make sure this pool is ready to be safely deleted from cephfs (ceph fs rm_data_pool archive fs.data.archive.frames)? --Mike [1]https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QHFOGEKXK7VDNNSKR74BA6IIMGGIXBXA/#7YQ6SSTESM5LTFVLQK3FSYFW5FDXJ5CF [2]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005233.h…

2 5

Remapped PGs
by David Orman 02 Mar '21

02 Mar '21

Hi, We see that we have 5 'remapped' PGs, but are unclear why/what to do about it. We shifted some target ratios for the autobalancer and it resulted in this state. When adjusting ratio, we noticed two OSDs go down, but we just restarted the container for those OSDs with podman, and they came back up. Here's status output: ################### root@ceph01:~# ceph status INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w) mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl, ceph02.gcglcg, ceph04.smbdew, ceph05.yropto osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs data: pools: 3 pools, 1057 pgs objects: 18.00M objects, 69 TiB usage: 119 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 1056 active+clean 1 active+clean+scrubbing+deep io: client: 859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr root@ceph01:~# ################### When I look at ceph pg dump, I don't see any marked as remapped: ################### root@ceph01:~# ceph pg dump |grep remapped INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 dumped all root@ceph01:~# ################### Any idea what might be going on/how to recover? All OSDs are up. Health is 'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman 2.0.3.

2 4

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

Default data pool in CEPH
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi, I have a CEPH 15.2.4 running in a docker. How to configure for use a specific data pool? i try put the follow line in the ceph.conf but the changes not working. . [client.myclient] rbd default data pool = Mydatapool I need it to configure for erasure pool with cloudstack Can help me? , where is the ceph conf we i need configure? Thanks. -- Untitled Document

4 6

Re: NFS Ganesha NFSv3
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi Thanks for the reply. cephadm runs ceph containers automatically. How to set privileged mode in ceph container? -- > El 23/9/20 a las 13:24, Daniel Gryniewicz escribió: >> NFSv3 needs privileges to connect to the portmapper. Try running >> your docker container in privileged mode, and see if that helps. >> >> Daniel >> >> On 9/23/20 11:42 AM, Gabriel Medve wrote: >>> Hi, >>> >>> I have a CEPH 15.2.5 running in a docker , i configure nfs ganesha >>> with nfs version 3 but i can not mount it. >>> If configure ganesha with nfs version 4 i can mounted without >>> problems but i need the version 3 . >>> >>> The error is mount.nfs: Protocol not supported >>> >>> Can help me? >>> >>> Thanks. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- > Untitled Document

2 1

Possible to disable check: x pool(s) have no replicas configured
by Marc Roos 23 Feb '21

23 Feb '21

Is it possible to disable checking on 'x pool(s) have no replicas configured', so I don't have this HEALTH_WARN constantly. Or is there some other disadvantage of keeping some empty 1x replication test pools?

2 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

radosgw bucket index issue
by Fox, Kevin M 02 Feb '21

02 Feb '21

We have a fairly old cluster that has over time been upgraded to nautilus. We were digging through some things and found 3 bucket indexes without a corresponding bucket. They should have been deleted but somehow were left behind. When we try and delete the bucket index, it will not allow it as the bucket is not found. The bucket index list command works fine though without the bucket. Is there a way to delete the indexes? Maybe somehow relink the bucket so it can be deleted again? Thanks, Kevin

1 1

Planning: Ceph User Survey 2020
by Mike Perez 28 Jan '21

28 Jan '21

Hi everyone, The Ceph User Survey 2020 is being planned by our working group. Please review the draft survey pdf, and let's discuss any changes. You may also join us in the next meeting *on November 25th at 12pm *PT https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group https://tracker.ceph.com/attachments/download/5260/Ceph%20User%20Survey%202… We're aiming to have something ready by mid-December. -- Mike Perez he/him Ceph / Rook / RDO / Gluster Community Architect Open-Source Program Office (OSPO) M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

5 6

PG inconsistent with empty inconsistent objects
by Seena Fallah 17 Jan '21

17 Jan '21

Hi, I'm facing something strange! One of the PGs in my pool got inconsistent and when I run `rados list-inconsistent-obj $PG_ID --format=json-pretty` the `inconsistents` key was empty! What is this? Is it a bug in Ceph or..? Thanks.

3 4

Ceph 15.2.3 on Ubuntu 20.04 with odroid xu4 / python thread Problem
by Dominik H 16 Jan '21

16 Jan '21

Hi all, i try to run ceph client tools on an odroid xu4 (armhf) with Ubuntu 20.04 on python 3.8.5. Unfortunately there is the following error on each "ceph" command (even in ceph --help) Traceback (most recent call last): File "/usr/bin/ceph", line 1275, in <module> retval = main() File "/usr/bin/ceph", line 981, in main cluster_handle = run_in_thread(rados.Rados, File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1342, in run_in_thread raise Exception("timed out") Exception: timed out With this Server I access an existing Ceph-Cluster with the same hardware. I checked the code part, there is just a thread start and a join (waiting for finish a RadosThread). Maybe this is a python error in combination with armhf architecture? Maybe someone can help. Thanks and greetings Dominik

3 8

Larger number of OSDs, cheroot, cherrypy, limits + containers == broken
by David Orman 08 Jan '21

08 Jan '21

Hi, We have a ceph 15.2.7 deployment using cephadm under podman w/ systemd. We've run into what we believe is: https://github.com/ceph/ceph-container/issues/1748 https://tracker.ceph.com/issues/47875 In our case, eventually the mgr container stops emitting output/logging. We are polling with external prometheus clusters, which is likely what triggers the issue, as it appears some amount of time after the container is spawned. Unfortunately, setting limits in the systemd service file for the mgr service on the host OS doesn't work, nor does modifying the unit.run file which is used to start the container under podman to include the --ulimit settings as suggested. Looking inside the container: lib/systemd/system/ceph-mgr@.service:LimitNOFILE=1048576 This prevents us from deploying medium to large ceph clusters, so I would argue it's a high priority bug that should not be closed, unless there is a workaround that works until EPEL 8 contains the fixed version of cheroot and the ceph containers include it. My understanding is this was fixed in cheroot 8.4.0: https://github.com/cherrypy/cheroot/issues/249 https://github.com/cherrypy/cheroot/pull/301 Thank you in advance for any suggestions, David

4 10

radosgw sync using more capacity on secondary than on master zone
by Scheurer François 06 Jan '21

06 Jan '21

Dear Ceph contributors While our (new) rgw secondary zone is doing the initial data sync from our master zone, we noticed that the reported capacity usage was getting higher than on primary zone: Master Zone: ceph version 14.2.5 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "false", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 269868.9823989868, "size_utilized GiB": 17180102071.868008, <= (cf. other issue below) "num_objects": 100218788, "Compression Rate": 63660.899148714125 pool stats: => "stored": 403834132234240, "objects": 191530416, "kb_used": 692891724288, "bytes_used": 709521125670912, "percent_used": 0.7570806741714478, "max_avail": 136595529269248 erasure-code-profile: crush-device-class=nvme crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 16384 default rgw compression: no Secondary Zone: ceph version 14.2.12 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "true", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 65282.37313461304, "size_utilized GiB": 60779.72828538809, "num_objects": 23074921, "Compression Rate": 0.9310281683550253 pool stats: => "stored": 407816305115136, "objects": 118637638, "kb_used": 497822635396, "bytes_used": 509770378645504, "percent_used": 0.7275146245956421, "max_avail": 152744706965504 erasure-code-profile: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: EC k=8 m=2 bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 4096 default rgw compression: yes As you see, the secondary zone is using 408 TB vs 404 TB on master zone. Summing size_actual for each buckets gives only 65 TB vs 270 TB on master zone. Any idea about what could cause such a difference? Is it a known issue? The are some known issue with Space Overhead with EC Pools for alloc_size > 4 KiB, cf. : https://www.mail-archive.com/ceph-users@ceph.io/msg06191.html https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/OHPO43J54TPBEUISYC… https://www.spinics.net/lists/ceph-users/msg59587.html But our secondary zone is on nvme osd's with bluestore_min_alloc_size_ssd=4096, so that should be fine. I will also investigate further with "radosgw-admin bucket radoslist" and rgw-orphan-list. Thank you in advance for any help. And Happy New Year very soon ;-) Best Regards Francois PS: The value on master zone "size_utilized GiB": 17180102071.868008 is wrong. This is due to a bucket with wrong stats: { "bucket": "XXX", "tenant": "", "zonegroup": "d29ea82c-4f77-40af-952b-2ab0705ad268", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "cb1594b3-a782-49d0-a19f-68cd48870a63.3119460.1", "marker": "cb1594b3-a782-49d0-a19f-68cd48870a63.63841.1481", "index_type": "Normal", "owner": "e13d054f6e9c4eea881f687923d7d380", "ver": "0#373075,1#286613,2#290913,3#341069,4#360862,5#341416,6#279526,7#352172,8#255944,9#314797,10#317650,11#305698,12#289557,13#345344,14#345273,15#294708,16#241001,17#298577,18#274866,19#293952,20#307635,21#334606,22#265355,23#302567,24#277505,25#307278,26#266963,27#297452,28#332274,29#319133,30#361027,31#314294,32#282887,33#324849,34#278560,35#307506,36#287269,37#344789,38#345389,39#323814,40#386483,41#280319,42#358072,43#336651,44#339176,45#248079,46#356784,47#381496,48#295152,49#251661,50#318661,51#330530,52#263564,53#332005,54#332937,55#320163,56#300485,57#296138,58#343271,59#359351,60#295711,61#275751,62#332264,63#351532", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0", "mtime": "2020-06-03 12:13:01.207610Z", "max_marker": "0#,1#,2#,3#00000341068.137793799.5,4#00000360861.156871075.5,5#00000341415.97799619.5,6#00000279525.155275260.5,7#,8#,9#00000314796.95564320.5,10#,11#,12#00000289556.137784091.5,13#,14#,15#,16#00000241000.126242121.5,17#,18#00000274865.124884405.5,19#,20#00000307634.137793798.5,21#00000334605.93836734.5,22#,23#00000302566.125226103.5,24#,25#,26#,27#00000297451.125229375.5,28#,29#00000319132.155275278.5,30#00000361026.98341455.5,31#,32#,33#00000324848.126242117.5,34#00000278559.124884417.5,35#,36#,37#00000344788.125945123.5,38#00000345388.97796715.5,39#,40#00000386482.98341457.5,41#,42#00000358071.124884415.5,43#,44#00000339175.135084366.5,45#00000248078.155263912.5,46#00000356783.98341461.5,47#00000381495.94538350.5,48#00000295151.138701826.5,49#00000251660.137793803.5,50#00000318660.93848186.5,51#,52#,53#00000332004.126242119.5,54#00000332936.138701824.5,55#,56#00000300484.156871073.5,57#,58#00000343270.98341459.5,59#00000359350.94257302.5,60#,61#,62#,63#00000351531.135084368.5", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551604 }, "rgw.main": { "size": 22615692324233, "size_actual": 22617004142592, "size_utilized": 18446736471689615156, "size_kb": 22085637036, "size_kb_actual": 22086918108, "size_kb_utilized": 18014391085634390, "num_objects": 521927 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 977 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } This is another issue, probably not related to our sync issue. -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

2 2

logging to stdout/stderr causes huge container log file
by Tony Liu 05 Jan '21

05 Jan '21

Hi, With ceph 15.2.5 octopus, mon, mgd and rgw dump loggings on debug level to stdout/stderr. It causes huge container log file (/var/lib/docker/containers/<ID>/<ID>-json.log). Is there any way to stop dumping logs or change the logging level? BTW, I tried "ceph config set <service> log_to_stderr false". It doesn't help. Thanks! Tony

2 2

Bluestore migration: per-osd device copy
by Chris Dunlop 05 Jan '21

05 Jan '21

Hi, The docs have scant detail on doing a migration to bluestore using a per-osd device copy: https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-o… This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a copy function (all the way from v9 to current). Has anyone actually tried doing this? Is there any further detail available on what is involved, e.g. a broad outline of the steps? Of course, detailed instructions would be even better, even if accompanied by "here be dragons!" warnings. Cheers, Chris

3 6

What is the specific meaning "total_time" in RGW ops log
by opengers 04 Jan '21

04 Jan '21

Hello everyone，I enabled rgw ops log by setting "rgw_enable_ops_log = true"，There is a "total_time" field in rgw ops log But I want to figure out whether "total_time" includes the period of time when rgw returns a response to the client?

2 2

Sequence replacing a failed OSD disk?
by Rainer Krienke 04 Jan '21

04 Jan '21

Hello, hope you had a nice Xmas and I wish all of you a good and happy new year in advance... Yesterday my ceph nautilus 14.2.15 cluster had a disk with unreadable sectors, after several tries the OSD was marked down and rebalancing started and has also finished successfully. ceph osd stat shows the osd now as "autoout,exists". Usually the steps to replace a failed disk are: 1. Destroy the failed OSD: ceph osd destroy {id} 2. run ceph-volume lvm create --bluestore --osd-id {id} --data /dev/sdX ... with a new disk in place to recreate a OSD with the same id without the need to change the crushmap or auth info etc. Now I still wait for a new disk and I am a unsure if I should run the destroy-command already now to keep ceph from trying to reactivate the broken osd? Then I would wait until the disk has arrived in a day or so and then use ceph volume to create a new osd? Or should I leave the state as it is now until the disk has arrived and then run both steps (destroy, volume ceph-lvm-create) one right after the other? Do the two slightly different ways make any difference if for example a power failure would result in a reboot of the node with the failed OSD before I could replace the broken disk? Any comments on this? Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312

2 1

Re: kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)
by Marc Roos 04 Jan '21

04 Jan '21

Hi Eugen, Indeed some really useful tips explaining what goes wrong, yet this thread[1] is about cephfs directly mounted on the osd node. I was having this also quite some time without any problems until suddenly I ran into the same issue as they had. I think I did not have any issues with the kernel client cephfs mount with luminous until I enabled the cephfs snapshots. Then I had to switch to the fuse client. In my case I am running a vm on the the osd node, which I thought would be different. I have been able to reproduce this stale mount just 2 times now. I have been testing with 10x more clients and it still works. Anyway I decided to move everything to rbd. I have been running vm's with rbd images without problems colocated on osd nodes for quite some time. I really would like to use the hosts because they have each 16c/32t and an average load of just 2-3. Unfortunately I did not document precisely how I recovered from the stale mount. I would like to see if I can reduce the amount of steps to take. Things started happening for me after I did the mds failover. Then I got blocked clients that I could unblock and fix the mount with mount -l Thanks for the pointers, I have linked them in my docs ;) -----Original Message----- To: ceph-users(a)ceph.io Subject: [ceph-users] Re: kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production) Hi, there have been several threads about hanging cephfs mounts, one quite long thread [1] describes a couple of debugging options but also mentions to avoid mounting cephfs on OSD nodes in a production environment. Do you see blacklisted clients with 'ceph osd blacklist ls'? If the answer is yes try to unblock that client [2]. The same option ('umount -l') is available on a cephfs client, you can try that, too. Other options described in [1] are to execute an MDS failover, but sometimes a reboot of that VM is the only solution left. Regards, Eugen [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028719.html [2] https://docs.ceph.com/en/latest/cephfs/eviction/#advanced-un-blocklisting-a… Zitat von Marc Roos <M.Roos(a)f1-outsourcing.eu>: > Is there not some genius out there that can shed a ligth on this? ;) > Currently I am not able to reproduce this. Thus it would be nice to > have some procedure at hand that resolves stale cephfs mounts nicely. > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something > like umount -l available?) (help wanted going to production) > > > > I have a vm on a osd node (which can reach host and other nodes via > the macvtap interface (used by the host and guest)). I just did a > simple > bonnie++ test and everything seems to be fine. Yesterday however the > dovecot procces apparently caused problems (only using cephfs for an > archive namespace, inbox is on rbd ssd, fs meta also on ssd) > > How can I recover from such lock-up. If I have a similar situation > with an nfs-ganesha mount, I have the option to do a umount -l, and > clients recover quickly without any issues. > > Having to reset the vm, is not really an option. What is best way to > resolve this? > > > > Ceph cluster: 14.2.11 (the vm has 14.2.16) > > I have in my ceph.conf nothing special, these 2x in the mds section: > > mds bal fragment size max = 120000 > # maybe for nfs-ganesha problems? > # http://docs.ceph.com/docs/master/cephfs/eviction/ > #mds_session_blacklist_on_timeout = false > #mds_session_blacklist_on_evict = false mds_cache_memory_limit = > 17179860387 > > > All running: > CentOS Linux release 7.9.2009 (Core) > Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC > 2020 x86_64 x86_64 x86_64 GNU/Linux > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Data migration between clusters
by Szabo, Istvan (Agoda) 04 Jan '21

04 Jan '21

What is the easiest and best way to migrate bucket from an old cluster to a new one? Luminous to octopus not sure does it matter from the data perspective. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

OSD reboot loop after running out of memory
by Stefan Wild 02 Jan '21

02 Jan '21

Hi, We recently upgraded a cluster from 15.2.1 to 15.2.5. About two days later, one of the server ran out of memory for unknown reasons (normally the machine uses about 60 out of 128 GB). Since then, some OSDs on that machine get caught in an endless restart loop. Logs will just mention system seeing the daemon fail and then restarting it. Since the out of memory incident, we’ve have 3 OSDs fail this way at separate times. We resorted to wiping the affected OSD and re-adding it to the cluster, but it seems as soon as all PGs have moved to the OSD, the next one fails. This is also keeping us from re-deploying RGW, which was affected by the same out of memory incident, since cephadm runs a check and won’t deploy the service unless the cluster is in HEALTH_OK status. Any help would be greatly appreciated. Thanks, Stefan

5 17

High read IO on RocksDB/WAL since upgrade to Octopus
by Glen Baars 31 Dec '20

31 Dec '20

Hello Ceph Users, Since upgrading from Nautilus to Octopus ( cluster started in luminous ) I have been trying to debug why the RocksDB/WAL is maxing out the SSD drives. ( QD > 32, 12000 read IOPS, 200 write IOPS ). The omap upgrade on migration was disabled initially but I reenabled it and restarted all OSD's. This completed without issue. I have increased the memory target from 4 to 6GB per OSD but it doesn't look like it is using it all anyway ( based on top ). I have offline compacted all OSDs. This seems to help for about 4-6 hours ( backfilling is occuring - maybe this triggers it? ). RGW garbage collection is upto date. Pg_log on some PG's are high due to them not being in a clean state ( 8% PGs > 3000 ) remainder of PG's I have reduced to 500 logs - no change. I've been working on this issue for days not without much luck. Nothing in the logs indicates a major issue. The client impact is a major reduction in speed. { "mon": { "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 5 }, "mgr": { "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 1 }, "osd": { "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 18, "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 280 }, "mds": {}, "rgw": { "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 2 }, "tcmu-runner": { "ceph version 14.2.13-450-g65ea1b614d (65ea1b614db8b6d10f334a8ff67c4de97f73bcbf) nautilus (stable)": 2 }, "overall": { "ceph version 14.2.13-450-g65ea1b614d (65ea1b614db8b6d10f334a8ff67c4de97f73bcbf) nautilus (stable)": 2, "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 18, "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)": 288 } } Any assistance in debugging would be greatly helpful. Glen This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

2 1

Debian repo for ceph-iscsi
by Chris Palmer 30 Dec '20

30 Dec '20

I just went to setup an iscsi gateway on a Debian Buster / Octopus cluster and hit a brick wall with packages. I had perhaps naively assumed they were in with the rest. Now I understand that it can exist separately, but then so can RGW. I found some ceph-iscsi rpm builds for Centos, but nothing for Debian. Are they around somewhere? The prerequisite packages python-rtslib-2.1.fb68 and tcmu-runner-1.4.0 also don't seem to be readily available for Debian. Has anyone done this for Debian? Thanks, Chris

4 5

device management and failure prediction
by Suresh Rama 28 Dec '20

28 Dec '20

Dear All, Hope you all had a great Christmas and much needed time off with family! Have any of you used "*device management and failure prediction"* in Nautilus? If yes, what is your feedback? Do you use LOCAL or CLOUD prediction models? https://ceph.io/update/new-in-nautilus-device-management-and-failure-predic… Your feedback and input is valuable. -- Regards, Suresh

2 1

Nautilus Health Metrics
by DHilsbos＠performair.com 28 Dec '20

28 Dec '20

All; I turned on device health metrics in one of our Nautilus clusters. Unfortunately, it doesn't seem to be collecting any information. When I do "ceph device get-health-metrics <device>, I get the following; { "20200821-223626": { "dev": "/dev/sdc", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "samsung_ssd_860_evo_4tb", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" } } The cluster is Nautilus 14.2.16 (updated from 14.2.11 just after turning on health metrics). Smartctl is release 7.0 dated 2018-12-30 at 14:47:55 UTC. Thoughts? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

1 0

Copying data from OneFS source to CEPHFS both shared via SAMBA
by Oskari Koivisto 28 Dec '20

28 Dec '20

Hi, I have case where storage backend is about to change from OneFS to Ceph. Both are mainly for windows clients used as object storage. Both ends have SAMBA configured as a GW with integration to AD. Both of the backends work and can be accessed and utilized but the problem is the data-transfer in between. So far I have tested only robocopy but that would be a preferred tool since xfer is done via Windows machine mounting both source and target. The copying it self works, no problem but copying the ACL for the files and folders doesn’t. If I provide “robocopy source target /MIR /SEC” I get access denied, only target top folder is created and it has wrong previleges. The CEPHFS should be correctly mounted via SAMBA since I can create and copy files and folders when they are created by, in this case, the administrator. And I can change the permission after wards but this is not the way the copying is to be done. There is a lot of data and permissions are crucial to preserve. I’ve tried many ways to save the permissions, among robocopy I’ve tried saving the permissions with "icacls save/restore” and with powershells command also. None of those seem to work. Here’s a conclusion of what works: creating new folder/files with administrator to source and robocopying them to target with /MIR and /SEC. copying without /SEC. Here’s also what doesn’t work or help: Taking over permission for Administrator. Remove non-existent users from source folder/files. Removing in-heritage for the folder in source. Changing owner to administrator. The CEPH is Nautilus. Any ideas or suggestiong for the topic? Best regards and happy new year! -Oskari

1 0

To be able to post to Ceph-users mailing list
by EKTA SINGHAI 27 Dec '20

27 Dec '20

email id: esinghai(a)es.iitr.ac.in

1 0

Random heartbeat_map timed out
by Seena Fallah 26 Dec '20

26 Dec '20

Hi, All my OSD nodes in the SSD tier are getting heartbeat_map timed out randomly and I don't find why! 7ff2ed3f2700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7ff2c8943700' had timed out after 15 It occurs many times in a day and causes my cluster to be down. Is there any way to find why the OSDs get time out? I don't think it's because of heartbeat and there is an issue with OSD that came to the heartbeat to be timeout because ODSs don't suicide and OSDs get too slow and cause downtime on RBD and S3 gateway because the queue is full! Thanks.

2 3

post
by mariyam sofi 24 Dec '20

24 Dec '20

1 0

cephfs flags question
by Stefan Kooman 23 Dec '20

23 Dec '20

Hi List, In order te reproduce an issue we see on a production cluster (cephFS client: ceph-fuse outperform kernel client by a factor of 5) we would like to have a test cluster to have the same cephfs "flags" as production. However, it's not completely clear how certain features influence the cephfs flags. What I could find in the source code, cephfs_features.h, is that it *seems* to correspond to the Ceph release. For example CEPHFS_FEATURE_NAUTILUS gets a "12" as feature bit. An upgraded (Luminous -> Mimic -> Nautilus) cephfs gives us the following cephfs flags: "1c". A (newly installed) Nautilus cluster gives "10" when new snapshots are not allowed (ceph fs set cephfs allow_new_snaps false) and "12" when new snapshots are allowed (ceph fs set cephfs allow_new_snaps true). We would like to have the test cluster get the "1c" flags and see if we can reproduce the issue. How can we achieve that? Any info on how those cephfs flags are constructed is welcome. Thanks, Gr. Stefan

2 8

after octopus cluster reinstall, rbd map fails with timeout
by Philip Brown 23 Dec '20

23 Dec '20

More banging on my prototype cluster, and ran into an odd problem. Used to be, when I create an rbd device, then try to map it, it would initially fail, saying I have to disable some features. Then I just run the suggested disable line -- usually rbd feature disable poolname/rbdname object-map fast-diff deep-flatten and then I can map it fine. but now after the latest cluster recreation, when I try to map, I just get # rbd map testpool/zfs02 rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (110) Connection timed out and no errors in dmesg output if I try to disable those features anyway, I get librbd::Operations: one or more requested features are already disabled(22) Invalid argument nothing in /var/log/ceph/cephadm.log either Any suggestions? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 1

Recreate pool device_health_metrics
by Victor Rodriguez 23 Dec '20

23 Dec '20

Hello, TL;DR How can I recreate the device_health_metrics pool? I'm experimenting with Ceph Octopus v15.2.8 in a 3 node cluster under Proxmox 6.3. After initializing CEPH the usual way, a "device_health_metrics" pool is created as soon as I create the first manager. That pool has just 1 PG but no OSD assigned, as OSD have not been created yet. After creating a few OSD and waiting for a couple days, that PG is still in "stale+undersized+peered" state. So I thought I could just disable monitoring (ceph device monitoring off), delete that pool and create it again with something like: ceph osd pool create device_health_metrics 1 --autoscale-mode=off The issue is that after recreating the pool and enabling monitoring (ceph device monitoring on), I get no data stored in it regardind my devices, even after running a manual scraping with ceph device scrape-health-metrics. Thank you in advance. Victor

1 0

cephadm cluster issues
by Duncan Bellamy 23 Dec '20

23 Dec '20

Hi, I am trying to set up a new cluster with cephadm using a docker backend. The initial boot strap did not finish cleanly and it errored out waiting for the mon-ip, I used the command: cephadm bootstrap --mon-ip 192.168.0.1 With 192.168.0.1 being the ip address of this first host. I tried the command again but it failed as the new ceph node was actually running so it could not bind to the ports. After a bit of searching I was able to use "sudo cephadm shell —“ commands to change the username and password for the dashboard and login to it. I then used cephadm to add a new host with "sudo cephadm shell — ceph orch host add host2” Now in the inventory of the dashboard, and "ceph orch device ls” only devices on host2 are listed not host1. In the Cluster/Hosts section of the dashboard host1 has its root volume drive listed in devices, and host2 has the root volume drive and drive for the OSD listed. I successfully added an OSD with a drive on host2, trying the same command adjusted for host1 I get the following in the log: Dec 23 08:55:47 localhost systemd[1]: var-lib-docker-overlay2-91e9dffa86c333353dd6b445021c852d7ce8da6237d0d4d95909d68ef3d4fe23\x2dinit-merged.mount: Succeeded. Dec 23 08:55:47 localhost systemd[24638]: var-lib-docker-overlay2-91e9dffa86c333353dd6b445021c852d7ce8da6237d0d4d95909d68ef3d4fe23\x2dinit-merged.mount: Succeeded. Dec 23 08:55:47 localhost containerd[1470]: time="2020-12-23T08:55:47.369773808Z" level=info msg="shim containerd-shim started" address=/containerd-shim/80f876072532ebebdfef341a5c793654e27766f2d1708991a6f25599b24b6557.sock debug=false pid=28597 Dec 23 08:55:47 localhost bash[8745]: debug 2020-12-23T08:55:47.517+0000 ffff73d7a200 1 mon.host1(a)0(leader).osd e12 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 71303168 kv_alloc: 876609536 Dec 23 08:55:47 localhost containerd[1470]: time="2020-12-23T08:55:47.621748606Z" level=info msg="shim reaped" id=69a786e4a61605c1e6eca5a6e0e5ed0900635a214b0f1c96a4f26ea7911a12ff Dec 23 08:55:47 localhost dockerd[2930]: time="2020-12-23T08:55:47.631479207Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Dec 23 08:55:47 localhost systemd[24638]: var-lib-docker-overlay2-91e9dffa86c333353dd6b445021c852d7ce8da6237d0d4d95909d68ef3d4fe23-merged.mount: Succeeded. Dec 23 08:55:47 localhost systemd[1]: var-lib-docker-overlay2-91e9dffa86c333353dd6b445021c852d7ce8da6237d0d4d95909d68ef3d4fe23-merged.mount: Succeeded. Dec 23 08:55:47 localhost systemd[24638]: var-lib-docker-overlay2-64bb135bc0cdab187566992dc9870068dee1430062e1a2b484381c19e03da895\x2dinit-merged.mount: Succeeded. Dec 23 08:55:47 localhost systemd[1]: var-lib-docker-overlay2-64bb135bc0cdab187566992dc9870068dee1430062e1a2b484381c19e03da895\x2dinit-merged.mount: Succeeded. Dec 23 08:55:47 localhost containerd[1470]: time="2020-12-23T08:55:47.972437378Z" level=info msg="shim containerd-shim started" address=/containerd-shim/4a61d63e1f46722ffa7a950c31145d167c5c69087d003e5928a6aa3a4831f031.sock debug=false pid=28659 Dec 23 08:55:48 localhost bash[8745]: cluster 2020-12-23T08:55:46.892633+0000 mgr.host1.kkssvi (mgr.24098) 24278 : cluster [DBG] pgmap v24212: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 931 GiB / 932 GiB avail Dec 23 08:55:48 localhost bash[8756]: debug 2020-12-23T08:55:48.889+0000 ffff93573700 0 log_channel(cluster) log [DBG] : pgmap v24213: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 931 GiB / 932 GiB avail Dec 23 08:55:49 localhost bash[8756]: debug 2020-12-23T08:55:49.085+0000 ffff9056f700 0 log_channel(audit) log [DBG] : from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "host1:/dev/nvme0n1", "target": ["mon-mgr", ""]}]: dispatch Dec 23 08:55:49 localhost bash[8745]: debug 2020-12-23T08:55:49.085+0000 ffff71575200 0 mon.host1@0(leader) e2 handle_command mon_command({"prefix": "osd tree", "states": ["destroyed"], "format": "json"} v 0) v1 Dec 23 08:55:49 localhost bash[8745]: debug 2020-12-23T08:55:49.085+0000 ffff71575200 0 log_channel(audit) log [DBG] : from='mgr.24098 192.168.0.1:0/2486989775' entity='mgr.host1.kkssvi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch Dec 23 08:55:49 localhost bash[8756]: debug 2020-12-23T08:55:49.089+0000 ffff8ed6d700 0 log_channel(cephadm) log [INF] : Found osd claims -> {} Dec 23 08:55:49 localhost bash[8756]: debug 2020-12-23T08:55:49.089+0000 ffff8ed6d700 0 log_channel(cephadm) log [INF] : Found osd claims for drivegroup None -> {} Dec 23 08:55:49 localhost containerd[1470]: time="2020-12-23T08:55:49.331868093Z" level=info msg="shim reaped" id=780a38dd49fce4a823c4c3d834abdd1cc17bbe0c0aa4f2dd7caeddf8dce1708e Dec 23 08:55:49 localhost dockerd[2930]: time="2020-12-23T08:55:49.341765820Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Dec 23 08:55:49 localhost systemd[24638]: var-lib-docker-overlay2-64bb135bc0cdab187566992dc9870068dee1430062e1a2b484381c19e03da895-merged.mount: Succeeded. Dec 23 08:55:49 localhost systemd[1]: var-lib-docker-overlay2-64bb135bc0cdab187566992dc9870068dee1430062e1a2b484381c19e03da895-merged.mount: Succeeded. Dec 23 08:55:49 localhost bash[8745]: audit 2020-12-23T08:55:49.091014+0000 mon.host1 (mon.0) 1093 : audit [DBG] from='mgr.24098 192.168.0.1:0/2486989775' entity='mgr.host1.kkssvi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch Dec 23 08:55:50 localhost bash[8745]: cluster 2020-12-23T08:55:48.893433+0000 mgr.host1.kkssvi (mgr.24098) 24279 : cluster [DBG] pgmap v24213: 1 pgs: 1 undersized+peered; 0 B data, 112 KiB used, 931 GiB / 932 GiB avail Dec 23 08:55:50 localhost bash[8745]: audit 2020-12-23T08:55:49.087597+0000 mgr.host1.kkssvi (mgr.24098) 24280 : audit [DBG] from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "host1:/dev/nvme0n1", "target": ["mon-mgr", ""]}]: dispatch Dec 23 08:55:50 localhost bash[8745]: cephadm 2020-12-23T08:55:49.093552+0000 mgr.host1.kkssvi (mgr.24098) 24281 : cephadm [INF] Found osd claims -> {} Dec 23 08:55:50 localhost bash[8745]: cephadm 2020-12-23T08:55:49.093933+0000 mgr.host1.kkssvi (mgr.24098) 24282 : cephadm [INF] Found osd claims for drivegroup None -> {} The other problem is logging is set to debug for both hosts, I tried "sudo cephadm shell -- ceph daemon mon.host1 config set mon_cluster_log_file_level info” which reports success but logging remains at debug level. If I try the same command with mon.host2 I get INFO:cephadm:Inferring fsid ae111111-1111-1111-1111-f1111a11111a INFO:cephadm:Inferring config /var/lib/ceph/ae147088-4486-11eb-9044-f1337a55707a/mon.host1/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 admin_socket: exception getting command descriptions: [Errno 2] No such file or directory Which looks like it is trying to use the config for host1 on host2? Thanks, Duncan

1 1

krbd cache quesitions
by huxiaoyu＠horebdata.cn 23 Dec '20

23 Dec '20

Dear ceph folks, rbd_cache can be set up as a read /write cache for librbd, widely used with openstack cinder. Does krbd has a silmilar cache controll mechanism or not? I am using krbd for iSCSI and NFS backend storage, and wonder whether a cache setting exists for krbd. thanks in advance, Samuel huxiaoyu(a)horebdata.cn

2 1

Ceph rgw & dashboard problem
by Mika Saari 23 Dec '20

23 Dec '20

Hi, Using Ceph Octopus installed with cephadm here. Version running currently is 15.2.6. There are 3 machines running the cluster. Machine names are introduced in /etc/hosts in long(FQDN) & short forms but ceph hostnames of the servers are in short form (not sure if this affects anyway). rdb side is working nicely, tested with a linux client. Trying to get object gateway to be visible in dashboard but getting error when selecting "Object Gateway -> Daemons" Error: RGW REST API failed request with status code 403 (b'{"Code":"AccessDenied","RequestId":"tx000000000000000000040-005fe20384-8ecbc' b'-ou","HostId":"8ecbc-ou-default"}') What am I doing wrong here? Thanks a lot, -Mika ---- Procedure what I have done ---- 1) ceph orch apply rgw default ou --placement="1 ceph1" 2) radosgw-admin user create --uid=test --display-name=test --access-key=test --secret-key=test 3) radosgw-admin period update --rgw-realm=default --commit 4) aws configure --profile=default aws configure --profile=default AWS Access Key ID [None]: test AWS Secret Access Key [None]: test Default region name [None]: default Default output format [None]: json 5) aws s3 mb s3://test1 --endpoint-url http://ceph1 make_bucket: test1 5.1) radosgw-admin bucket list [ "test1" ] 6) ceph dashboard --help | grep reset-rgw | awk '{print $2}' | xargs -n 1 ceph dashboard Option RGW_API_ACCESS_KEY reset to default value "" Option RGW_API_ADMIN_RESOURCE reset to default value "admin" Option RGW_API_HOST reset to default value "" Option RGW_API_PORT reset to default value "80" Option RGW_API_SCHEME reset to default value "http" Option RGW_API_SECRET_KEY reset to default value "" Option RGW_API_SSL_VERIFY reset to default value "True" Option RGW_API_USER_ID reset to default value "" 7) ceph dashboard set-rgw-api-user-id "test" Option RGW_API_USER_ID updated 8) ceph dashboard set-rgw-api-access-key test Option RGW_API_ACCESS_KEY updated 9) ceph dashboard set-rgw-api-secret-key test Option RGW_API_SECRET_KEY updated 10) ceph mgr module disable dashboard 11) ceph mgr module enable dashboard

2 2

diskprediction_local fails with python3-sklearn 0.22.2
by Eric Dold 22 Dec '20

22 Dec '20

Hello the mgr module diskprediction_local fails under ubuntu 20.04 focal with python3-sklearn version 0.22.2 Ceph version is 15.2.3 when the module is enabled i get the following error: File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in serve self.predict_all_devices() File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in predict_all_devices result = self._predict_life_expentancy(devInfo['devid']) File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in _predict_life_expentancy predicted_result = obj_predictor.predict(predict_datas) File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457, in predict pred = clf.predict(ordered_data) File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in predict if self.break_ties and self.decision_function_shape == 'ovo': AttributeError: 'SVC' object has no attribute 'break_ties' Best Regards Eric

2 1

Failing OSD RocksDB Corrupt
by Ashley Merrick 22 Dec '20

22 Dec '20

Hello,I had some faulty power cables on some OSD's in one server which caused lots of IO issues/disks appearing/disappearing.This has been corrected now, 2 of the 10 OSD's are working, however 8 are failing to start due to what looks to be a corrupt DB.When running a ceph-bluestore-tool fsck I get the following output:rocksdb: [db/db_impl_open.cc:516] db.wal/002221.log: dropping 1302 bytes; Corruption: missing start of fragmented record(2)2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: [db/db_impl.cc:389] Shutdown: canceling all background work2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: [db/db_impl.cc:563] Shutdown complete2020-12-22T16:21:52.715+0100 7f7b6a1500c0 -1 rocksdb: Corruption: missing start of fragmented record(2)2020-12-22T16:21:52.715+0100 7f7b6a1500c0 -1 bluestore(/var/lib/ceph/b1db6b36-0c4c-4bce-9cda-18834be0632d/osd.28) opendb erroring opening db:Trying to start the OSD leads to:ceph_abort_msg("Bad table magic number: expected 9863518390377041911, found 9372993859750765257 in db/002442.sst")It looks like the last write to these OSD's never fully completed, sadly as I was adding this new node to move from OSD to Host redundancy (EC Pool) I have 20% down PG's currently, is there anything I can do to remove the last entry in the DB or somehow clean up the rocksDB to get these OSD's atleast started? Understand may end up with some corrupted files.Thanks Sent via MXlogin

1 0

PGs down
by Jeremy Austin 22 Dec '20

22 Dec '20

I could use some input from more experienced folks… First time seeing this behavior. I've been running ceph in production (replicated) since 2016 or earlier. This, however, is a small 3-node cluster for testing EC. Crush map rules should sustain the loss of an entire node. Here's the EC rule: rule cephfs425 { id 6 type erasure min_size 3 max_size 6 step set_chooseleaf_tries 40 step set_choose_tries 400 step take default step choose indep 3 type host step choose indep 2 type osd step emit } I had actual hardware failure on one node. Interestingly, this appears to have resulted in data loss. OSDs began to crash in a cascade on other nodes (i.e., nodes with no known hardware failure). Not a low RAM problem. I could use some pointers about how to get the down PGs back up — I *think* there are enough EC shards, even disregarding the OSDs that crash on start. nautilus 14.2.15 ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 54.75960 root default -10 16.81067 host sumia 1 hdd 5.57719 osd.1 up 1.00000 1.00000 5 hdd 5.58469 osd.5 up 1.00000 1.00000 6 hdd 5.64879 osd.6 up 1.00000 1.00000 -7 16.73048 host sumib 0 hdd 5.57899 osd.0 up 1.00000 1.00000 2 hdd 5.56549 osd.2 up 1.00000 1.00000 3 hdd 5.58600 osd.3 up 1.00000 1.00000 -3 21.21844 host tower1 4 hdd 3.71680 osd.4 up 0 1.00000 7 hdd 1.84799 osd.7 up 1.00000 1.00000 8 hdd 3.71680 osd.8 up 1.00000 1.00000 9 hdd 1.84929 osd.9 up 1.00000 1.00000 10 hdd 2.72899 osd.10 up 1.00000 1.00000 11 hdd 3.71989 osd.11 down 0 1.00000 12 hdd 3.63869 osd.12 down 0 1.00000 cluster: id: d0b4c175-02ba-4a64-8040-eb163002cba6 health: HEALTH_ERR 1 MDSs report slow requests 4/4239345 objects unfound (0.000%) Too many repaired reads on 3 OSDs Reduced data availability: 7 pgs inactive, 7 pgs down Possible data damage: 4 pgs recovery_unfound Degraded data redundancy: 95807/24738783 objects degraded (0.387%), 4 pgs degraded, 3 pgs undersized 7 pgs not deep-scrubbed in time 7 pgs not scrubbed in time services: mon: 3 daemons, quorum sumib,tower1,sumia (age 4d) mgr: sumib(active, since 7d), standbys: sumia, tower1 mds: cephfs:1 {0=sumib=up:active} 2 up:standby osd: 13 osds: 11 up (since 3d), 10 in (since 4d); 3 remapped pgs data: pools: 5 pools, 256 pgs objects: 4.24M objects, 15 TiB usage: 24 TiB used, 24 TiB / 47 TiB avail pgs: 2.734% pgs not active 95807/24738783 objects degraded (0.387%) 47910/24738783 objects misplaced (0.194%) 4/4239345 objects unfound (0.000%) 245 active+clean 7 down 3 active+recovery_unfound+undersized+degraded+remapped 1 active+recovery_unfound+degraded+repair progress: Rebalancing after osd.12 marked out [============================..] Rebalancing after osd.4 marked out [=============================.] An snipped from an example down pg: "up": [ 3, 2, 5, 1, 8, 9 ], "acting": [ 3, 2, 5, 1, 8, 9 ], <snip> ], "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 11, 12 ], "peering_blocked_by": [ { "osd": 11, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 12, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] }, { Oddly, these OSDs possibly did NOT experience hardware failure. However, they won't start -- see pastebin for ceph-osd.11.log https://pastebin.com/6U6sQJuJ HEALTH_ERR 1 MDSs report slow requests; 4/4239345 objects unfound (0.000%); Too many repaired reads on 3 OSDs; Reduced data availability : 7 pgs inactive, 7 pgs down; Possible data damage: 4 pgs recovery_unfound; Degraded data redundancy: 95807/24738783 objects degraded (0 .387%), 4 pgs degraded, 3 pgs undersized; 7 pgs not deep-scrubbed in time; 7 pgs not scrubbed in time MDS_SLOW_REQUEST 1 MDSs report slow requests mdssumib(mds.0): 42 slow requests are blocked > 30 secs OBJECT_UNFOUND 4/4239345 objects unfound (0.000%) pg 19.5 has 1 unfound objects pg 15.2f has 1 unfound objects pg 15.41 has 1 unfound objects pg 15.58 has 1 unfound objects OSD_TOO_MANY_REPAIRS Too many repaired reads on 3 OSDs osd.9 had 9664 reads repaired osd.7 had 9665 reads repaired osd.4 had 12 reads repaired PG_AVAILABILITY Reduced data availability: 7 pgs inactive, 7 pgs down pg 15.10 is down, acting [3,2,5,1,8,9] pg 15.1e is down, acting [5,1,9,8,2,3] pg 15.40 is down, acting [7,10,1,5,3,2] pg 15.4a is down, acting [0,3,5,6,9,10] pg 15.6a is down, acting [3,2,6,1,10,8] pg 15.71 is down, acting [3,2,1,6,8,10] pg 15.76 is down, acting [2,0,6,5,10,9] PG_DAMAGED Possible data damage: 4 pgs recovery_unfound pg 15.2f is active+recovery_unfound+undersized+degraded+remapped, acting [5,1,0,3,2147483647,7], 1 unfound pg 15.41 is active+recovery_unfound+undersized+degraded+remapped, acting [5,1,0,3,2147483647,2147483647], 1 unfound pg 15.58 is active+recovery_unfound+undersized+degraded+remapped, acting [10,2147483647,2,3,1,5], 1 unfound pg 19.5 is active+recovery_unfound+degraded+repair, acting [3,2,5,1,8,10], 1 unfound PG_DEGRADED Degraded data redundancy: 95807/24738783 objects degraded (0.387%), 4 pgs degraded, 3 pgs undersized pg 15.2f is stuck undersized for 635305.932075, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,1,0,3,2147483647,7] pg 15.41 is stuck undersized for 364298.836902, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,1,0,3,2147483647,2147483647] pg 15.58 is stuck undersized for 384461.110229, current state active+recovery_unfound+undersized+degraded+remapped, last acting [10,2147483647,2,3,1,5] pg 19.5 is active+recovery_unfound+degraded+repair, acting [3,2,5,1,8,10], 1 unfound PG_NOT_DEEP_SCRUBBED 7 pgs not deep-scrubbed in time pg 15.76 not deep-scrubbed since 2020-10-21 14:30:03.935228 pg 15.71 not deep-scrubbed since 2020-10-21 12:20:46.235792 pg 15.6a not deep-scrubbed since 2020-10-21 07:52:33.914083 pg 15.10 not deep-scrubbed since 2020-10-22 03:24:40.465367 pg 15.1e not deep-scrubbed since 2020-10-22 10:37:36.169959 pg 15.40 not deep-scrubbed since 2020-10-23 05:33:35.208748 pg 15.4a not deep-scrubbed since 2020-10-22 05:14:06.981035 PG_NOT_SCRUBBED 7 pgs not scrubbed in time pg 15.76 not scrubbed since 2020-10-24 08:12:40.090831 pg 15.71 not scrubbed since 2020-10-25 05:22:40.573572 pg 15.6a not scrubbed since 2020-10-24 15:03:09.189964 pg 15.10 not scrubbed since 2020-10-24 16:25:08.826981 pg 15.1e not scrubbed since 2020-10-24 16:05:03.080127 pg 15.40 not scrubbed since 2020-10-24 11:58:04.290488 pg 15.4a not scrubbed since 2020-10-24 11:32:44.573551 -- Jeremy Austin jhaustin(a)gmail.com

4 14

osd_pglog memory hoarding - another case
by Kalle Happonen 22 Dec '20

22 Dec '20

Hello all, wrt: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7IMIWCKIHXN… Yesterday we hit a problem with osd_pglog memory, similar to the thread above. We have a 56 node object storage (S3+SWIFT) cluster with 25 OSD disk per node. We run 8+3 EC for the data pool (metadata is on replicated nvme pool). The cluster has been running fine, and (as relevant to the post) the memory usage has been stable at 100 GB / node. We've had the default pg_log of 3000. The user traffic doesn't seem to have been exceptional lately. Last Thursday we updated the OSDs from 14.2.8 -> 14.2.13. On Friday the memory usage on OSD nodes started to grow. On each node it grew steadily about 30 GB/day, until the servers started OOM killing OSD processes. After a lot of debugging we found that the pg_logs were huge. Each OSD process pg_log had grown to ~22GB, which we naturally didn't have memory for, and then the cluster was in an unstable situation. This is significantly more than the 1,5 GB in the post above. We do have ~20k pgs, which may directly affect the size. We've reduced the pg_log to 500, and started offline trimming it where we can, and also just waited. The pg_log size dropped to ~1,2 GB on at least some nodes, but we're still recovering, and have a lot of ODSs down and out still. We're unsure if version 14.2.13 triggered this, or if the osd restarts triggered this (or something unrelated we don't see). This mail is mostly to figure out if there are good guesses why the pg_log size per OSD process exploded? Any technical (and moral) support is appreciated. Also, currently we're not sure if 14.2.13 triggered this, so this is also to put a data point out there for other debuggers. Cheers, Kalle Happonen

4 16

kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)
by Marc Roos 22 Dec '20

22 Dec '20

I have a vm on a osd node (which can reach host and other nodes via the macvtap interface (used by the host and guest)). I just did a simple bonnie++ test and everything seems to be fine. Yesterday however the dovecot procces apparently caused problems (only using cephfs for an archive namespace, inbox is on rbd ssd, fs meta also on ssd) How can I recover from such lock-up. If I have a similar situation with an nfs-ganesha mount, I have the option to do a umount -l, and clients recover quickly without any issues. Having to reset the vm, is not really an option. What is best way to resolve this? Ceph cluster: 14.2.11 (the vm has 14.2.16) I have in my ceph.conf nothing special, these 2x in the mds section: mds bal fragment size max = 120000 # maybe for nfs-ganesha problems? # http://docs.ceph.com/docs/master/cephfs/eviction/ #mds_session_blacklist_on_timeout = false #mds_session_blacklist_on_evict = false mds_cache_memory_limit = 17179860387 All running: CentOS Linux release 7.9.2009 (Core) Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

2 4

Can big data use Ceph?
by fantastic2085 22 Dec '20

22 Dec '20

Can big data use Ceph?For example, can Hive Hbase Spark use Ceph? https://github.com/ceph/cephfs-hadoop is no longer maintain?

4 3

friendly warning about death by container versions
by Philip Brown 22 Dec '20

22 Dec '20

I was having horrible problems getting my test ceph clusterj reinitialized. All kinds of annoying things were happening. including things like getting differing output from ceph orch device ls vs ceph device ls Being new-ish to ceph, i was going nuts, wondering what kind of init options I was missing. Turns out, nothing i was doing was wrong, per se. I had ended up with differing container versions. Even after doing "cephadm rm-cluster", the old versions were sticking. The really annoying thing is, the difference was tiny. 15.2.8 on the master. 15.2.5 on other nodes. but device related things were failing, with 2020-12-21 09:37:09,134 INFO /bin/podman:stderr ceph-volume inventory: error: unrecognized arguments: --filter-for-batch in /var/log/ceph/cephadm.log Sighhh... To save anyone else some research, the end-user fix is: run ceph orch upgrade start --ceph-version 15.2.8 -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 0

Cephfs mount hangs
by Marc Roos 21 Dec '20

21 Dec '20

How to recover from this? It is possible to have a vm with a cephfs mount on a osd server?

1 0

guide to multi-homed hosts, for Octopus?
by Philip Brown 21 Dec '20

21 Dec '20

In my experiments with ceph so far, setting up a new cluster goes fairly well... so long as i only use a single network. But when I try to use separate networks, things stop functioning in various ways. (For example, I can " SO I thought I'd ask for pointers to any multi-network setup guide. My goal: * have a 3+ node ceph cluster that each has local SSD storage only. * have an RBD mapped on each node, which will then have a non-cephfs filesystem on it, shared out via NFS * have each node share out NFS on one interface, but communicate to the cluster on a separate interface In the old ways, it was theoretically straightforward, in that you could specify "public" interfaces vs other ones. But in the new cephadm driven world, I havent found the magic that works. For example, in my current iteration, I have successfully added all three hosts, and have 3 "mon"s... but "ceph orch device ls --refresh" only shows the dev from the node I'm running it on. Im still trying to cycle through different bootstrap options. and Im experimenting with overriding naming in /etc/hosts, for things like which IP addresses get mapped to the real hostname, vs which get given "hostname-datainterface" type naming. For example: On the one hand, Im wondering if I need to name ALL IP addresses for a host, with the same hostname. But on the other hand, my sysadmin instincts whisper to me that sounds like a terrible idea. SO, tips from people who have done multi homing under octopus, would be appreciated. Note that my initial proof-of-concept cluster is just 3 physical nodes, so everything needs to live on them. -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 0

Ceph with Firewall
by Seena Fallah 21 Dec '20

21 Dec '20

Hi, I want to enable the firewall on my ceph nodes with ufw. Does anyone have any experience with any performance regression in it? Is there any solution for blocking exporter ports without a firewall in a Ceph cluster like node exporter and ceph exporter? Thanks.

1 0

mgr's stop responding, dropping out of cluster with _check_auth_rotating
by Welby McRoberts 21 Dec '20

21 Dec '20

Hi Folks We've noticed that in a cluster of 21 nodes (5 mgrs&mons & 504 OSDs with 24 per node) that the mgr's are, after a non specific period of time, dropping out of the cluster. The logs only show the following: debug 2020-12-10T02:02:50.409+0000 7f1005840700 0 log_channel(cluster) log [DBG] : pgmap v14163: 4129 pgs: 4129 active+clean; 10 GiB data, 31 TiB used, 6.3 PiB / 6.3 PiB avail debug 2020-12-10T03:20:59.223+0000 7f10624eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-12-10T02:20:59.226159+0000) debug 2020-12-10T03:21:00.223+0000 7f10624eb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-12-10T02:21:00.226310+0000) The _check_auth_rotating repeats approximately every second. The instances are all syncing their time with NTP and have no issues on that front. A restart of the mgr fixes the issue. It appears that this may be related to https://tracker.ceph.com/issues/39264. The suggestion seems to be to disable prometheus metrics, however, this obviously isn't realistic for a production environment where metrics are critical for operations. Please let us know what additional information we can provide to assist in resolving this critical issue. Cheers Welby

4 6

Re: Setting up NFS with Octopus
by Jens Hyllegaard (Soft Design A/S) 21 Dec '20

21 Dec '20

I am sorry, but I am not sure how to do that? We have just started working with Ceph. -----Original Message----- From: Eugen Block <eblock(a)nde.ag> Sent: 18. december 2020 12:06 To: Jens Hyllegaard (Soft Design A/S) <jens.hyllegaard(a)softdesign.dk> Subject: Re: [ceph-users] Re: Setting up NFS with Octopus Oh you're right, it worked for me, I just tried that with a new path and it was created for me. Can you share the client keyrings? I have two nfs daemons running and they have these permissions: client.nfs.ses7-nfs.host2 key: AQClNNJf5KHVERAAAzhpp9Mclh5wplrcE9VMkQ== caps: [mon] allow r caps: [osd] allow rw pool=nfs-test namespace=ganesha client.nfs.ses7-nfs.host3 key: AQCqNNJf4rlqBhAARGTMkwXAldeprSYgmPEmJg== caps: [mon] allow r caps: [osd] allow rw pool=nfs-test namespace=ganesha Zitat von "Jens Hyllegaard (Soft Design A/S)" <jens.hyllegaard(a)softdesign.dk>: > On the Create NFS export page it says the directory will be created. > > Regards > > Jens > > > -----Original Message----- > From: Eugen Block <eblock(a)nde.ag> > Sent: 18. december 2020 11:52 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Setting up NFS with Octopus > > Hi, > > is the path (/objstore) present within your CephFS? If not you need to > mount the CephFS root first and create your directory to have NFS > access it. > > > Zitat von "Jens Hyllegaard (Soft Design A/S)" > <jens.hyllegaard(a)softdesign.dk>: > >> Hi. >> >> We are completely new to Ceph, and are exploring using it as an NFS >> server at first and expand from there. >> >> However we have not been successful in getting a working solution. >> >> I have set up a test environment with 3 physical servers, each with >> one OSD using the guide at: >> https://docs.ceph.com/en/latest/cephadm/install/ >> >> I created a new replicated pool: >> ceph osd pool create objpool replicated >> >> And then I deployed the gateway: >> ceph orch apply nfs objstore objpool nfs-ns >> >> I then created a new CephFS volume: >> ceph fs volume create objstore >> >> So far so good 😊 >> >> My problem is when I try to create the NFS export The settings are as >> follows: >> Cluster: objstore >> Daemons: nfs.objstore >> Storage Backend: CephFS >> CephFS User ID: admin >> CephFS Name: objstore >> CephFS Path: /objstore >> NFS Protocol: NFSV3 >> Access Type: RW >> Squash: all_squash >> Transport protocol: both UDP & TCP >> Client: Any client can access >> >> However when I click on Create NFS export, I get: >> Failed to create NFS 'objstore:/objstore' >> >> error in mkdirs /objstore: Permission denied [Errno 13] >> >> Has anyone got an idea as to why this is not working? >> >> If you need any further information, do not hesitate to say so. >> >> >> Best regards, >> >> Jens Hyllegaard >> Senior consultant >> Soft Design >> Rosenkaeret 13 | DK-2860 Søborg | Denmark | +45 39 66 02 00 | >> softdesign.dk<http://www.softdesign.dk/> | synchronicer.com >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

2 8

Is there a command to update a client with a new generated key?
by Marc Roos 21 Dec '20

21 Dec '20

Is there a command to update a client with a new generated key? Something like: ceph auth new-key client.rbd Could be usefull if you accidentaly did a ceph auth ls, because that still displays keys ;)

2 2

python API
by Vladimir Prokofev 21 Dec '20

21 Dec '20

Hello. I'm trying to write some Python code for analysis of my RBD images storage usage, rbd and rados package versions are 14.2.16. Basically I want the same data that I can acquire from shell 'rbd du <image>' and 'rbd info <image>' commands, but through Python API. At the moment I can connect to the cluster, get an image list from the pool, but that's about it. As far as I can read, there's no API method to obtain image info such as provisioned/used storage, etc. Is this really not possible with the current Python API, or am I missing something? Documentation used: https://docs.ceph.com/en/latest/rbd/api/librbdpy/ https://docs.ceph.com/en/latest/rados/api/python/

1 0

Issues upgrading from ceph 15.2.7 to 15.2.8 related to cephadm pull
by David Orman 20 Dec '20

20 Dec '20

Hi, We have an Octopus installation using cephadm/containers in podman. We're trying to upgrade from 15.2.7 to 15.2.8. The mgrs upgrade successfully, but the we then get a failure that cephadm failed to pull an image: [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image Failed to pull docker.io/ceph/ceph:v15.2.8 on ceph02: Looking at this server, and trying to run the cephadm pull manually: root@ceph02:~# cephadm pull Using recent ceph image docker.io/ceph/ceph:v15.2.8 Pulling container image docker.io/ceph/ceph:v15.2.8... Traceback (most recent call last): File "/usr/sbin/cephadm", line 6111, in <module> r = args.func() File "/usr/sbin/cephadm", line 1381, in _infer_image return func() File "/usr/sbin/cephadm", line 2676, in command_pull return command_inspect_image() File "/usr/sbin/cephadm", line 1381, in _infer_image return func() File "/usr/sbin/cephadm", line 2716, in command_inspect_image info_from = get_image_info_from_inspect(out.strip(), args.image) File "/usr/sbin/cephadm", line 2727, in get_image_info_from_inspect image_id, digests = out.split(',', 1) ValueError: not enough values to unpack (expected 2, got 1) root@ceph02:~# cephadm version Using recent ceph image docker.io/ceph/ceph:v15.2.8 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable) root@ceph02:~# podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/ceph/ceph v15.2.8 5553b0cb212c 3 days ago 965 MB docker.io/ceph/ceph v15.2.7 2bc420ddb175 2 weeks ago 979 MB docker.io/ceph/ceph v15 2bc420ddb175 2 weeks ago 979 MB docker.io/prom/node-exporter v0.18.1 e5a616e4b9cf 18 months ago 24.3 MB root@ceph02:~# We've upgraded this cluster multiple times prior without issue, so this appear to be unique to the 15.2.7 -> 15.2.8 upgrade. If we pass --image with docker.io/ceph/ceph:v15.2.8, the same behavior occurs. It seems like cephadm in the 15.2.8 image isn't playing nicely with the pull command, for some reason. Interestingly, I did a fresh install of v15.2.8 on one server, and after the installation, trying a 'cephadm pull' also fails on that 15.2.8 setup, so it seems like the pull functionality may be broken on these images.

1 1

New cephadm install, OSD down after a few hours
by Jie Zhang 20 Dec '20

20 Dec '20

Hello, I'm new to ceph and setting up my first cluster and playing around with it. I followed the steps in cephadm guide ( https://docs.ceph.com/en/latest/cephadm/install/) Here is my implementation of the instructions, from a Ubuntu Server 20.04.1 install: sudo apt-get update && sudo apt-get dist-upgrade -y sudo apt-get install curl git nano docker docker-compose docker.io attr ntp bash-completion -y sudo usermod -aG docker $USER curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm chmod +x cephadm sudo ./cephadm add-repo --release octopus sudo ./cephadm install sudo mkdir -p /etc/ceph sudo cephadm install ceph-common ceph sudo cephadm bootstrap --mon-ip <IP> ceph orch daemon add server-node1:/dev/sda ceph orch daemon add server-node1:/dev/sdc ceph orch daemon add server-node1:/dev/sdd ceph orch daemon add server-node1:/dev/sde ceph orch daemon add server-node1:/dev/sdf Then I log into the ceph dashboard, change password, click around. Take a break... ...HOURS Later... My OSDs are down. I notice this in ceph-volume.log: [ceph_volume.main][INFO ] Running command: ceph-volume lvm deactivate 1 7a0fb8df-2d12-4e96-9def-5b2c195f6af4 So I run: ceph-volume lvm activate --all And my OSDs are back and 'cephadm ls' shows them as legacy. However, running 'cephadm adopt --style legacy --name osd.0' causes the osd to go down again. What is going on? PS: The only two other issues I in the logs see are /usr/bin/docker:stderr Error: No such object: ceph-<ID>-osd.0 and [ceph_volume.util.system][INFO ] /var/lib/ceph/osd/ceph-0 does not appear to be a tmpfs mount Jie

1 0

failed to process reshard logs
by Seena Fallah 19 Dec '20

19 Dec '20

Hi, I used radosgw-admin reshard process to process a manual bucket resharding after it completes it logs an error below ERROR: failed to process reshard logs, error=(2) No such file or directory I've added a bucket to resharding queue with radosgw-admin reshard add --bucket bucket-tmp --num-shards 2053 Is anything wrong with it? Using nautilus 14.2.14. Thanks.

1 0

Clearing contents of OSDs without removing them?
by Dallas Jones 19 Dec '20

19 Dec '20

Stumbling closer toward a usable production cluster with Ceph, but I have yet another stupid n00b question I'm hoping you all will tolerate. I have 38 OSDs up and in across 4 hosts. I (maybe prematurely) removed my test filesystem as well as the metadata and data pools used by the deleted filesystem. This leaves me with 38 OSDs with a bunch of data on them. Is there a simple way to just whack all of the data on all of those OSDs before I create new pools and a new filesystem? Version: ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) As you can see from the partial output of ceph -s, I left a bunch of crap spread across the OSDs... pools: 8 pools, 32 pgs objects: 219 objects, 1.2 KiB usage: 45 TiB used, 109 TiB / 154 TiB avail pgs: 32 active+clean Thanks in advance for a shove in the right direction. -Dallas

3 2

Fwd: ceph-fuse false passed X_OK check
by Alex Taylor 19 Dec '20

19 Dec '20

cc to the list On Thu, Dec 17, 2020 at 11:39 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote: > > On Wed, Dec 16, 2020 at 5:46 PM Alex Taylor <alexu4993(a)gmail.com> wrote: > > > > Hi Cephers, > > > > I'm using VSCode remote development with a docker server. It worked OK > > but fails to start the debugger after /root mounted by ceph-fuse. The > > log shows that the binary passes access X_OK check but cannot be > > actually executed. see: > > > > ``` > > strace_log: access("/root/.vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7", > > X_OK) = 0 > > > > root@develop:~# ls -alh > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > -rw-r--r-- 1 root root 978 Dec 10 13:06 > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > ``` > > > > I also test the access syscall on ext4, xfs and even cephfs kernel > > client, all of them return -EACCES, which is expected (the extension > > will then explicitly call chmod +x). > > > > After some digging in the code, I found it is probably caused by > > https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L5549-L5550. > > So here come two questions: > > 1. Is this a bug or is there any concern I missed? > > I tried reproducing it with the master branch and could not. It might > be due to an older fuse/ceph. I suggest you upgrade! > I tried the master(332a188d9b3c4eb5c5ad2720b7299913c5a772ee) as well and the issue still exists. My test program is: ``` #include <stdio.h> #include <unistd.h> int main() { int r; const char path[] = "test"; r = access(path, F_OK); printf("file exists: %d\n", r); r = access(path, X_OK); printf("file executable: %d\n", r); return 0; } ``` And the test result: ``` # local filesystem: ext4 root@f626800a6e85:~# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:13 test root@f626800a6e85:~# ./a.out file exists: 0 file executable: -1 root@f626800a6e85:~# findmnt -t fuse.ceph-fuse TARGET SOURCE FSTYPE OPTIONS /root/mnt ceph-fuse fuse.ceph-fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other root@f626800a6e85:~# cd mnt # ceph-fuse root@f626800a6e85:~/mnt# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:10 test root@f626800a6e85:~/mnt# ./a.out file exists: 0 file executable: 0 root@f626800a6e85:~/mnt# ./test bash: ./test: Permission denied ``` Again, ceph-fuse says file `test` is executable but in fact it can't be executed. The kernel version I'm testing on is: ``` root@f626800a6e85:~/mnt# uname -ar Linux f626800a6e85 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux ``` Please try the program above and make sure you're running it as root user, thank you. And if the reproduction still fails, please let me know the kernel version. > > 2. It works again with fuse_default_permissions=true, any drawbacks if > > this option is set? > > Correctness (ironically, for you) and performance. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >

1 0

OSD spec db_devices: rotational: 0 not working in 15.2.7
by David Orman 18 Dec '20

18 Dec '20

Hi, We deployed a cluster with an OSD spec like this: service_type: osd service_id: osd_spec_test1 placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 This worked on an earlier version of 15 (I think we deployed with 15.2.5). However, this same drive spec results in the following in ceph orch ls --export: --- service_type: osd service_id: osd_spec_test1 service_name: osd.osd_spec_test1 placement: host_pattern: '*' spec: data_devices: rotational: 1 db_slots: 12 filter_logic: AND objectstore: bluestore Now, if we change rotational to 1 for db_devices: service_type: osd service_id: osd_spec_test1 service_name: osd.osd_spec_test1 placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 1 db_slots: 12 filter_logic: AND objectstore: bluestore For some reason, db_devices is ignored if rotational: 0 (our intended use case). This definitely worked before, our entire cluster was provisioned this way. We ran into this when attempting to replace a broken OSD. Any ideas what might be going on?

1 0

Change MON IP in containerized environment
by Eugen Block 18 Dec '20

18 Dec '20

Hi Cephers, there have been several threads in this mailing list about changing IP addresses (especially MON services) and the docs [1] are based on pre-container deployments. I tried to achieve the same for an Octopus cluster (with podman). The "right way" mentioned in the docs is quite easy so I focused on the "messy way". I wrote a blog post [2] about it, maybe it is helpful to some of you. Best regards, Eugen [1] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a… [2] https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors…

1 0

Setting up NFS with Octopus
by Jens Hyllegaard (Soft Design A/S) 18 Dec '20

18 Dec '20

Hi. We are completely new to Ceph, and are exploring using it as an NFS server at first and expand from there. However we have not been successful in getting a working solution. I have set up a test environment with 3 physical servers, each with one OSD using the guide at: https://docs.ceph.com/en/latest/cephadm/install/ I created a new replicated pool: ceph osd pool create objpool replicated And then I deployed the gateway: ceph orch apply nfs objstore objpool nfs-ns I then created a new CephFS volume: ceph fs volume create objstore So far so good 😊 My problem is when I try to create the NFS export The settings are as follows: Cluster: objstore Daemons: nfs.objstore Storage Backend: CephFS CephFS User ID: admin CephFS Name: objstore CephFS Path: /objstore NFS Protocol: NFSV3 Access Type: RW Squash: all_squash Transport protocol: both UDP & TCP Client: Any client can access However when I click on Create NFS export, I get: Failed to create NFS 'objstore:/objstore' error in mkdirs /objstore: Permission denied [Errno 13] Has anyone got an idea as to why this is not working? If you need any further information, do not hesitate to say so. Best regards, Jens Hyllegaard Senior consultant Soft Design Rosenkaeret 13 | DK-2860 Søborg | Denmark | +45 39 66 02 00 | softdesign.dk<http://www.softdesign.dk/> | synchronicer.com

2 1

Erasure Space not showing on Octopus
by Jeremi Avenant 18 Dec '20

18 Dec '20

Good day I currently have a problem where my octopus cluster shows cephfs EC free space differently from my luminous cluster cephfs EC datapool. The only difference I notice is the get application per pool. Mounting the volume from a vm in production Luminous 12.2.13 EC3+1: 10.102.25.18:6789,10.102.25.19:6789,10.102.25.28:6789:/volumes/_nogroup/6f485332-da3d-4f6d-b5aa-a68e9566d1dc 2.8P 2.5P 327T 89% /ilifu 2.8P EC available space 2.5P EC used space 327T available on the EC cephfs_data ceph osd pool application get cephfs_data { "cephfs": {} } Mounting the volume from a vm in production Octopus 15.2.8 EC8+2: 10.102.36.3:6789,10.102.36.5:6789,10.102.36.7:6789:/volumes/_nogroup/<longno> 4.3P 20T 4.3P 1% /new 4.3P is the RAW space of the whole of the ceph cluster, I'm expecting 3.0P Usable Space as show on ceph df cephfs_data 14 16 422 B 3 422 B 0 3.0 PiB ceph osd pool application get cephfs_data { "cephfs": { "data": "cephfs" } } If anyone has an idea how to get the EC volume to show the EC space and not the raw space. Thank you -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

1 0

v14.2.15 Nautilus released
by David Galloway 18 Dec '20

18 Dec '20

This is the 15th backport release in the Nautilus series. This release fixes a ceph-volume regression introduced in v14.2.13 and includes few other fixes. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-15-nautilus-released Notable Changes --------------- * ceph-volume: Fixes lvm batch --auto, which breaks backward compatibility when using non rotational devices only (SSD and/or NVMe). * BlueStore: Fixes a bug in collection_list_legacy which makes pgs inconsistent during scrub when running mixed versions of osds, prior to 14.2.12 with newer. * MGR: progress module can now be turned on/off, using the commands: `ceph progress on` and `ceph progress off`. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.15.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: afdd217ae5fb1ed3f60e16bd62357ca58cc650e5

4 4

Bucket operations an issue with C# AWSSDK.S3 client
by Szabo, Istvan (Agoda) 18 Dec '20

18 Dec '20

Dear Community, We are having issues with bucket operations with ceph octopus 15.2.7. The client library used is : AWSSDK.S3 Version 3.5.6.6 Also used an old version of client: AWSSDK Version 2.3.55.2 Used it in both .net core and simple .net projects but the same result # Note : All methods mentioned below work perfectly fine with nodejs client Node js Client: https://www.npmjs.com/package/ceph Nodejs client doc: https://github.com/YounGoat/nodejs.osapi/blob/2f9d82092589bb50e452c57131499… # ALSO Note : All methods mentioned below work alright with another old version of ceph (luminous 12.2.8) with the same c# client Here are the details: The client object below is new AmazonS3Client( "<Our_Access_Key>", "<Our_Secret_Key>", new AmazonS3Config {ServiceURL = "<Our_Service_Url>" } ); 1. List Buckets. Result: OK Library method used : client.ListBucketsAsync(); [cid:image001.png@01D6D465.D7957510] 1. Create a New Bucket Result: Error 405 MethodNotAllowed (You might think it's a permission issue but it's not, because the same creds work in nodejs) Library method used : var request = new PutBucketRequest {BucketName = "seo"}; // seo is an existing bucket created by command line client.PutBucketAsync(request); Result error snapshot: [cid:image002.png@01D6D465.D7957510] 1. Create Object in an existing bucket Result: Error 501 Not Implemented Library method used : var request = new PutObjectRequest { BucketName = "seo", Key = "test2", ContentType = "text/plain", ContentBody = value, }; client.PutObjectAsync(request); We tried adding headers for content length as well but does not work as well Result error snapshot: [cid:image003.png@01D6D465.D7957510] 1. Read object existing in a bucket Existing bucket name: seo Existing object name(key): test Result: No Such Bucket Library code method: var request = new GetObjectRequest {BucketName = "seo", Key = "test"}; client.GetObjectAsync(request); Result error snapshot: [cid:image004.png@01D6D465.D7957510] Thank you in advance your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 1

changing OSD IP addresses in octopus/docker environment
by Philip Brown 18 Dec '20

18 Dec '20

I was wondering how to change the IPs used for the OSD servers, in my new Octopus based environment, which uses all those docker/podman images by default. imiting date range to within a year, doesnt seem to hit anything. unlimited google search pulled up http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/020503.h… but that references editing /etc/ceph/ceph.conf and changing the [osd.x] sections. which dont exist with Octopus, as far as I've found so far. It doesnt exist in the top level host's /etc/ceph/ceph.conf nor does it exist in the container's conf file, as viewed via "cephadm shell" So, what are the options here? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 1

MDS Corruption: ceph_assert(!p) in MDCache::add_inode
by Brandon Lyon 18 Dec '20

18 Dec '20

This is attempt #3 to submit this issue to this mailing list. I don't expect this to be received. I give up. I have an issue with MDS corruption which so far I haven't been able to resolve using the recovery steps I've found online. I'm on v15.2.6. I've tried all the recovery steps mentioned here, except copying the pool: https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ When I try to start an MDS instance, it crashes after a few seconds. It logs a bunch of "bad backtrace on directory inode" errors before failing on an assertion in MDCache::add_inode, line 313: https://github.com/ceph/ceph/blob/cb8c61a60551b72614257d632a574d420064c17a/… Here's the output of journalctl -xe: https://pastebin.com/9g1UJaKQ I asked in the IRC channel, and it was suggested I might be able to manually delete the duplicate inodes using the RADOS API, though I don't know specifically how I would do that. I have also cloned the code and built Ceph with the problem assertion replaced with a return, but I haven't tried using it yet and I'm saving that as my last resort. I'd appreciate any help you all can give. Thank you, - Brandon Lyon

1 0

bug? cant turn off rbd cache?
by Philip Brown 17 Dec '20

17 Dec '20

Oops.. sent this to the "wrong" list previously. (lists.ceph.com) Lets try the proper one this time :-/ not sure if this an actual bug or I'm doing something else wrong. but in Octopus, I have on the master node # ceph --version ceph version 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) octopus (stable) # tail -3 /etc/ceph/ceph.conf [client] rbd cache = false rbd cache writethrough until flush = false but I have restarted ALL nodes.. and yet rbd cache is still on. # ceph --admin-daemon `find /var/run/ceph -name 'ceph-mon*'` config show |grep rbd_cache "rbd_cache": "true", What else am I supposed to do??? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

3 8

performance degredation every 30 seconds
by Philip Brown 17 Dec '20

17 Dec '20

I have a new 3 node octopus cluster, set up on SSDs. I'm running fio to benchmark the setup, with fio --filename=/dev/rbd0 --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --numjobs=1 --time_based --group_reporting --name=iops-test-job --runtime=120 --eta-newline=1 However, I notice that, approximately every 30 seconds, performance tanks for a bit. Any ideas on why, and better yet, how to get rid of the problem? Sample debug output below. Notice the transitions at [eta 01m:27s] and [eta 00m:49s] It happens again at [00m:09], but figured I didnt need to redundantly post that. Jobs: 1 (f=1): [m(1)][2.5%][r=43.4MiB/s,w=43.3MiB/s][r=11.1k,w=11.1k IOPS][eta 01m:58s] Jobs: 1 (f=1): [m(1)][4.1%][r=47.3MiB/s,w=47.8MiB/s][r=12.1k,w=12.2k IOPS][eta 01m:56s] Jobs: 1 (f=1): [m(1)][5.8%][r=48.6MiB/s,w=49.3MiB/s][r=12.5k,w=12.6k IOPS][eta 01m:54s] Jobs: 1 (f=1): [m(1)][7.4%][r=52.4MiB/s,w=53.1MiB/s][r=13.4k,w=13.6k IOPS][eta 01m:52s] Jobs: 1 (f=1): [m(1)][9.1%][r=54.7MiB/s,w=54.1MiB/s][r=13.0k,w=13.8k IOPS][eta 01m:50s] Jobs: 1 (f=1): [m(1)][10.7%][r=41.5MiB/s,w=42.6MiB/s][r=10.6k,w=10.9k IOPS][eta 01m:48s] Jobs: 1 (f=1): [m(1)][12.4%][r=51.5MiB/s,w=50.6MiB/s][r=13.2k,w=12.0k IOPS][eta 01m:46s] Jobs: 1 (f=1): [m(1)][14.0%][r=16.6MiB/s,w=16.0MiB/s][r=4248,w=4098 IOPS][eta 01m:44s] Jobs: 1 (f=1): [m(1)][14.9%][r=33.3MiB/s,w=33.5MiB/s][r=8526,w=8579 IOPS][eta 01m:43s] Jobs: 1 (f=1): [m(1)][16.5%][r=47.1MiB/s,w=47.4MiB/s][r=12.1k,w=12.1k IOPS][eta 01m:41s] Jobs: 1 (f=1): [m(1)][18.2%][r=49.6MiB/s,w=49.0MiB/s][r=12.7k,w=12.8k IOPS][eta 01m:39s] Jobs: 1 (f=1): [m(1)][19.8%][r=50.3MiB/s,w=51.4MiB/s][r=12.9k,w=13.1k IOPS][eta 01m:37s] Jobs: 1 (f=1): [m(1)][21.5%][r=53.5MiB/s,w=52.9MiB/s][r=13.7k,w=13.5k IOPS][eta 01m:35s] Jobs: 1 (f=1): [m(1)][23.1%][r=52.7MiB/s,w=52.1MiB/s][r=13.5k,w=13.3k IOPS][eta 01m:33s] Jobs: 1 (f=1): [m(1)][24.8%][r=55.3MiB/s,w=54.9MiB/s][r=14.1k,w=14.1k IOPS][eta 01m:31s] Jobs: 1 (f=1): [m(1)][26.4%][r=44.0MiB/s,w=45.2MiB/s][r=11.5k,w=11.6k IOPS][eta 01m:29s] Jobs: 1 (f=1): [m(1)][28.1%][r=12.1MiB/s,w=11.8MiB/s][r=3105,w=3011 IOPS][eta 01m:27s] Jobs: 1 (f=1): [m(1)][29.8%][r=16.6MiB/s,w=17.3MiB/s][r=4238,w=4422 IOPS][eta 01m:25s] Jobs: 1 (f=1): [m(1)][31.4%][r=9820KiB/s,w=9516KiB/s][r=2455,w=2379 IOPS][eta 01m:23s] Jobs: 1 (f=1): [m(1)][33.1%][r=6974KiB/s,w=7099KiB/s][r=1743,w=1774 IOPS][eta 01m:21s] Jobs: 1 (f=1): [m(1)][34.7%][r=49.5MiB/s,w=49.2MiB/s][r=12.7k,w=12.6k IOPS][eta 01m:19s] Jobs: 1 (f=1): [m(1)][36.4%][r=49.3MiB/s,w=49.8MiB/s][r=12.6k,w=12.8k IOPS][eta 01m:17s] Jobs: 1 (f=1): [m(1)][38.0%][r=36.4MiB/s,w=35.9MiB/s][r=9326,w=9200 IOPS][eta 01m:15s] Jobs: 1 (f=1): [m(1)][39.7%][r=43.4MiB/s,w=43.3MiB/s][r=11.1k,w=11.1k IOPS][eta 01m:13s] Jobs: 1 (f=1): [m(1)][41.3%][r=47.1MiB/s,w=47.1MiB/s][r=12.1k,w=12.1k IOPS][eta 01m:11s] Jobs: 1 (f=1): [m(1)][43.0%][r=47.9MiB/s,w=48.0MiB/s][r=12.3k,w=12.5k IOPS][eta 01m:09s] Jobs: 1 (f=1): [m(1)][44.6%][r=49.9MiB/s,w=48.8MiB/s][r=12.8k,w=12.5k IOPS][eta 01m:07s] Jobs: 1 (f=1): [m(1)][46.3%][r=46.4MiB/s,w=46.9MiB/s][r=11.9k,w=11.0k IOPS][eta 01m:05s] Jobs: 1 (f=1): [m(1)][47.9%][r=46.7MiB/s,w=46.4MiB/s][r=11.0k,w=11.9k IOPS][eta 01m:03s] Jobs: 1 (f=1): [m(1)][49.6%][r=55.3MiB/s,w=55.3MiB/s][r=14.1k,w=14.2k IOPS][eta 01m:01s] Jobs: 1 (f=1): [m(1)][51.2%][r=54.1MiB/s,w=53.2MiB/s][r=13.8k,w=13.6k IOPS][eta 00m:59s] Jobs: 1 (f=1): [m(1)][52.9%][r=53.4MiB/s,w=52.9MiB/s][r=13.7k,w=13.6k IOPS][eta 00m:57s] Jobs: 1 (f=1): [m(1)][54.5%][r=58.8MiB/s,w=58.0MiB/s][r=15.1k,w=15.1k IOPS][eta 00m:55s] Jobs: 1 (f=1): [m(1)][56.2%][r=60.0MiB/s,w=58.6MiB/s][r=15.4k,w=15.0k IOPS][eta 00m:53s] Jobs: 1 (f=1): [m(1)][57.9%][r=57.7MiB/s,w=58.1MiB/s][r=14.8k,w=14.9k IOPS][eta 00m:51s] Jobs: 1 (f=1): [m(1)][59.5%][r=14.0MiB/s,w=14.3MiB/s][r=3592,w=3651 IOPS][eta 00m:49s] Jobs: 1 (f=1): [m(1)][61.2%][r=17.4MiB/s,w=17.4MiB/s][r=4443,w=4457 IOPS][eta 00m:47s] Jobs: 1 (f=1): [m(1)][62.8%][r=18.1MiB/s,w=18.7MiB/s][r=4640,w=4783 IOPS][eta 00m:45s] Jobs: 1 (f=1): [m(1)][64.5%][r=7896KiB/s,w=8300KiB/s][r=1974,w=2075 IOPS][eta 00m:43s] Jobs: 1 (f=1): [m(1)][66.1%][r=47.8MiB/s,w=47.3MiB/s][r=12.2k,w=12.1k IOPS][eta 00m:41s] -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

5 16

reliability of rados_stat() function
by Peter Lieven 17 Dec '20

17 Dec '20

Hi all, the rados_stat() function has a TODO in the comments: * TODO: when are these set, and by whom? can they be out of date? Can anyone help with this? How reliably is the pmtime updated? Is there a minimum update interval? Thank you, Peter

1 1

Namespace usability for mutitenancy
by George Shuklin 17 Dec '20

17 Dec '20

Hello. Had been someone starting using namespaces for real production for multi-tenancy? How good is it at isolating tenants from each other? Can they see each other presence, quotas, etc? Is is safe to give access via cephx to (possibly hostile to each other) users to the same pool with restrictions 'user per namespace'? How badly can one user affect others? Quotas restrict space overuse, but what about IO and omaps overuse?

1 0

who's managing the cephcsi plugin?
by Marc Roos 17 Dec '20

17 Dec '20

Is this cephcsi plugin under control of redhat?

1 0

allocate_bluefs_freespace failed to allocate / ceph_abort_msg("bluefs enospc")
by Stephan Austermühle 17 Dec '20

17 Dec '20

Hi all, in various Rook operated Ceph clusters I have seen OSDs going into a CrashLoop due to debug 2020-12-16T13:19:25.500+0000 7fc4c3f13f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1608124765507105, "job": 1, "event": "recovery_started", "log_files": [1400, 1402]} debug 2020-12-16T13:19:25.500+0000 7fc4c3f13f40 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #1400 mode 0 debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40 1 bluefs _allocate failed to allocate 0x43ce43d on bdev 1, free 0x2e50000; fallback to bdev 2 debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40 1 bluefs _allocate unable to allocate 0x43ce43d on bdev 2, free 0xffffffffffffffff; fallback to slow device expander debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40 -1 bluestore(/var/lib/ceph/osd/ceph-1) allocate_bluefs_freespace failed to allocate on 0x3d1b0000 min_size 0x43d0000 > allocated total 0x300000 bluefs_shared_alloc_size 0x10000 allocated 0x300000 available 0x b019c8000 debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40 -1 bluefs _allocate failed to expand slow device to fit +0x43ce43d debug 2020-12-16T13:19:27.724+0000 7fc4c3f13f40 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x43ce43d /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.6/rpm/el8/BUILD/ceph-15.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7fc4c3f13f40 time 2020-12-16T13:19:27.731533+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.6/rpm/el8/BUILD/ceph-15.2.6/src/os/bluestore/BlueFS.cc: 2721: ceph_abort_msg("bluefs enospc") The OSD is not really full: # ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-1 inferring bluefs devices from bluestore path 1 : device size 0x18ffc00000 : own 0x[bffe10000~fffe0000] = 0xfffe0000 : using 0xfd190000(4.0 GiB) : bluestore has 0x82b360000(33 GiB) available Expanding DB/WAL... Expanding the underlying block device by just 1 gig followed by "ceph-bluestore-tool bluefs-bdev-expand" and "ceph-bluestore-tool repair" resolves the situation. In general, larger OSDs seem to reduce the likeliness for this issue. Ceph version is v15.2.6. Is this a known bug? Ceph report and logs are attached. Thanks for your help Stephan

2 2

Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"
by Darrin Hodges 17 Dec '20

17 Dec '20

Hi all, Have an issue with my three monitors, they keep getting " e3 handle_auth_request failed to assign global_id" errors, subsequently, commands like 'ceph status' just hang. Any ideas on what the errors means? many thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

4 4

ceph-fuse false passed X_OK check
by Alex Taylor 17 Dec '20

17 Dec '20

Hi Cephers, I'm using VSCode remote development with a docker server. It worked OK but fails to start the debugger after /root mounted by ceph-fuse. The log shows that the binary passes access X_OK check but cannot be actually executed. see: ``` strace_log: access("/root/.vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7", X_OK) = 0 root@develop:~# ls -alh .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 -rw-r--r-- 1 root root 978 Dec 10 13:06 .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 ``` I also test the access syscall on ext4, xfs and even cephfs kernel client, all of them return -EACCES, which is expected (the extension will then explicitly call chmod +x). After some digging in the code, I found it is probably caused by https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L5549-L5550. So here come two questions: 1. Is this a bug or is there any concern I missed? 2. It works again with fuse_default_permissions=true, any drawbacks if this option is set?

2 2

v15.2.4 Octopus released
by David Galloway 17 Dec '20

17 Dec '20

We're happy to announce the fourth bugfix release in the Octopus series. In addition to a security fix in RGW, this release brings a range of fixes across all components. We recommend that all Octopus users upgrade to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-4-octopus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * Cephadm: There were a lot of small usability improvements and bug fixes: * Grafana when deployed by Cephadm now binds to all network interfaces. * `cephadm check-host` now prints all detected problems at once. * Cephadm now calls `ceph dashboard set-grafana-api-ssl-verify false` when generating an SSL certificate for Grafana. * The Alertmanager is now correctly pointed to the Ceph Dashboard * `cephadm adopt` now supports adopting an Alertmanager * `ceph orch ps` now supports filtering by service name * `ceph orch host ls` now marks hosts as offline, if they are not accessible. * Cephadm can now deploy NFS Ganesha services. For example, to deploy NFS with a service id of mynfs, that will use the RADOS pool nfs-ganesha and namespace nfs-ns:: ceph orch apply nfs mynfs nfs-ganesha nfs-ns * Cephadm: `ceph orch ls --export` now returns all service specifications in yaml representation that is consumable by `ceph orch apply`. In addition, the commands `orch ps` and `orch ls` now support `--format yaml` and `--format json-pretty`. * Cephadm: `ceph orch apply osd` supports a `--preview` flag that prints a preview of the OSD specification before deploying OSDs. This makes it possible to verify that the specification is correct, before applying it. * RGW: The `radosgw-admin` sub-commands dealing with orphans -- `radosgw-admin orphans find`, `radosgw-admin orphans finish`, and `radosgw-admin orphans list-jobs` -- have been deprecated. They have not been actively maintained and they store intermediate results on the cluster, which could fill a nearly-full cluster. They have been replaced by a tool, currently considered experimental, `rgw-orphan-list`. * RBD: The name of the rbd pool object that is used to store rbd trash purge schedule is changed from "rbd_trash_trash_purge_schedule" to "rbd_trash_purge_schedule". Users that have already started using `rbd trash purge schedule` functionality and have per pool or namespace schedules configured should copy "rbd_trash_trash_purge_schedule" object to "rbd_trash_purge_schedule" before the upgrade and remove "rbd_trash_purge_schedule" using the following commands in every RBD pool and namespace where a trash purge schedule was previously configured:: rados -p <pool-name> [-N namespace] cp rbd_trash_trash_purge_schedule rbd_trash_purge_schedule rados -p <pool-name> [-N namespace] rm rbd_trash_trash_purge_schedule or use any other convenient way to restore the schedule after the upgrade. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7447c15c6ff58d7fce91843b705a268a1917325c -- David Galloway Systems Administrator, RDU Ceph Engineering IRC: dgalloway

4 5

Ceph Outage (Nautilus) - 14.2.11
by Suresh Rama 17 Dec '20

17 Dec '20

Dear All, We have a 38 node HP Apollo cluster with 24 3.7T Spinning disk and 2 NVME for journal. This is one of our 13 clusters which was upgraded from Luminous to Nautilus (14.2.11). When one of our openstack customers uses elastic search (they offer Logging as a Service) to their end users reported IO latency issues, our SME rebooted two nodes that he felt were doing memory leak. The reboot didn't help rather worsen the situation and he went ahead and recycled the entire cluster one node a time as to fix the slow ops reported by OSDs. This caused a huge issue and MONS were not able to withstand the spam and started crashing. 1) We audited the network (inspecting TOR, iperf, MTR) and nothing was indicating any issue but OSD logs were keep complaining about BADAUTHORIZER 2020-12-13 15:32:31.607 7fea5e3a2700 0 --1- 10.146.126.200:0/464096978 >> v1:10.146.127.122:6809/1803700 conn(0x7fea3c1ba990 0x7fea3c1bf600 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2020-12-13 15:32:31.607 7fea5e3a2700 0 --1- 10.146.126.200:0/464096978 >> v1:10.146.127.122:6809/1803700 conn(0x7fea3c1c1e20 0x7fea3c1bcdf0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2020-12-13 15:32:31.607 7fea5e3a2700 0 --1- 10.146.126.200:0/464096978 >> v1:10.146.127.122:6809/1803700 conn(0x7fea3c1ba990 0x7fea3c1bf600 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER 2) Made sure no clock skew and we use timesyncd. After taking out a couple of OSDs that were indicating slow in the ceph health response the situation didn't improve. After 3 days of troubleshooting we upgraded the MONS to 14.2.15 and seems the situation improved a little but still reporting 61308 slow ops which we really struggled to isolate with bad OSDs as moving a couple of them didn't improve. One of the MON(2) failed to join the cluster and always doing compact and never was able to join (see the size below). I suspect that could be because the key value store information between 1 and 3 is not up to date with 2. At times, we had to stop and start to compress to get a better response from Ceph MON (keeping them running in one single MON). root@pistoremon-as-c01:~# du -sh /var/lib/ceph/mon 391G /var/lib/ceph/mon root@pistoremon-as-c03:~# du -sh /var/lib/ceph/mon 337G /var/lib/ceph/mon root@pistoremon-as-c02:~# du -sh /var/lib/ceph/mon 13G /var/lib/ceph/mon cluster: id: bac20301-d458-4828-9dd9-a8406acf5d0f health: HEALTH_WARN noout,noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average 10969 pgs not deep-scrubbed in time 46 daemons have recently crashed 61308 slow ops, oldest one blocked for 2572 sec, daemons [mon.pistoremon-as-c01,mon.pistoremon-as-c03] have slow ops. mons pistoremon-as-c01,pistoremon-as-c03 are using a lot of disk space 1/3 mons down, quorum pistoremon-as-c01,pistoremon-as-c03 services: mon: 3 daemons, quorum pistoremon-as-c01,pistoremon-as-c03 (age 52m), out of quorum: pistoremon-as-c02 mgr: pistoremon-as-c01(active, since 2h), standbys: pistoremon-as-c03, pistoremon-as-c02 osd: 911 osds: 888 up (since 68m), 888 in flags noout,noscrub,nodeep-scrub rgw: 2 daemons active (pistorergw-as-c01, pistorergw-as-c02) task status: data: pools: 17 pools, 32968 pgs objects: 62.98M objects, 243 TiB usage: 748 TiB used, 2.4 PiB / 3.2 PiB avail pgs: 32968 active+clean io: client: 56 MiB/s rd, 95 MiB/s wr, 1.78k op/s rd, 4.27k op/s wr 3) When looking through ceph.log on the mon with tailf, I was getting a lot of different time stamp reported in the ceph logs in MON1 which is master. Confused on why the live log report various timestamps? stat,write 2166784~4096] snapc 0=[] ondisk+write+known_if_redirected e951384) initiated 2020-12-13 06:16:58.873964 currently delayed 2020-12-13 06:39:37.169504 osd.1224 (osd.1224) 325855 : cluster [WRN] slow request osd_op(client.461445583.0:8881223 1.16aa 1:55684db0:::rbd_data.9ede65fc7af15.0000000000000000:head [stat,write 3547136~4096] snapc 0=[] ondisk+write+known_if_redirected e951384) initiated 2020-12-13 06:16:59.082012 currently delayed 2020-12-13 06:39:37.169510 osd.1224 (osd.1224) 325856 : cluster [WRN] slow request osd_op(client.461445583.0:8881191 1.16aa 1:55684db0:::rbd_data.9ede65fc7af15.0000000000000000:head [stat,write 2314240~4096] snapc 0=[] ondisk+write+known_if_redirected e951384) initiated 2020-12-13 06:16:58.874031 currently delayed 2020-12-13 06:39:37.169513 osd.1224 (osd.1224) 325857 : cluster [WRN] slow request osd_op(client.461445583.0:8881224 1.16aa 1:55684db0:::rbd_data.9ede65fc7af15.0000000000000000:head [stat,write 3571712~8192] snapc 0=[] ondisk+write+known_if_redirected e951384) initiated 2020-12-13 06:16:59.082094 currently delayed ^C root@pistoremon-as-c03:~# date Tue Dec 15 20:12:02 UTC 2020 root@pistoremon-as-c03:~# timedatectl Local time: Tue 2020-12-15 20:12:04 UTC Universal time: Tue 2020-12-15 20:12:04 UTC RTC time: Tue 2020-12-15 20:12:04 Time zone: Etc/UTC (UTC, +0000) Network time on: yes NTP synchronized: yes RTC in local TZ: no 4) Keep seeing the error below and assume that's due to the size of the MON DB with high workload the clients are not able to fetch the mon map ? 2020-12-13 13:16:56.197 7f946cddc700 -1 monclient: get_monmap_and_config failed to get config 2020-12-13 13:17:00.613 7f946cddc700 -1 monclient: get_monmap_and_config failed to get config We are getting all sorts of errors with primary concern on how to get the slow ops down and improve the IO latency. MONs using the SSD for the storage with NOOP scheduler with nr_request (128). Please provide some guidance on all 4 issues reported above. Thank you for your valueable time. -- Regards, Suresh

4 4

The ceph balancer sets upmap items which violates my crushrule
by Manuel Lausch 17 Dec '20

17 Dec '20

The ceph balancer sets upmap items which violates my crushrule the rule: rule cslivebapfirst { id 0 type replicated min_size 2 max_size 4 step take csliveeubap-u01dc step chooseleaf firstn 2 type room step emit step take csliveeubs-u01dc step chooseleaf firstn 2 type room step emit } So my intention is, that the first two replicas are stored in the datacenter „csliveeubap-u01dc“ and the next two replicas are stored in the datacenter „csliveeubs-u01dc“ The cluster has 49152 PGs and 665 of them has at least 3 replicas in one datacenter which is not expected! One example on PG 3.96e The acting OSDs are in this order: 504 -> DC: csliveeubap-u01dc, room: csliveeubap-u01r03 1968 -> DC: csliveeubap-u01dc, room: csliveeubap-u01r01 420 -> DC: csliveeubap-u01dc, room: csliveeubap-u01r02 1945 -> DC: csliveeubs-u01dc, room: csliveeubs-u01r01 This PG has one upmap item: ceph osd dump | grep 3.96e pg_upmap_items 3.96e [2013,420] OSD 2013 is in the DC: csliveeubs-u01dc I checked this by hand with ceph osd pg-upmap-item If I try to set two relicas in one room I will get a appropriate error in the mon log and nothing happens. But setting it to a other dc worked unfortunately. I would suggest this is a ugly bug. What do you think? ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) Manuel

3 2

[OSSN-0087] Ceph user credential leakage to consumers of OpenStack Manila
by gouthampravi＠gmail.com 17 Dec '20

17 Dec '20

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello, Forwarding a security note that was shared with the OpenStack community here for your awareness. This concerns a security vulnerability that has now been addressed. I'd like to thank Ceph contributors: Patrick Donnelly, Kotresh Hiremath Ravishankar and Ramana Raja for their help in addressing this issue. Please find information regarding patches and releases in the security note below. Thanks, Goutham Ceph user credential leakage to consumers of OpenStack Manila - ------------------------------------------------------------- ### Summary ### OpenStack Manila users can request access on a share to any arbitrary cephx user, including privileged pre-existing users of a Ceph cluster. They can then retrieve access secret keys for these pre-existing ceph users via Manila APIs. A cephx client user name and access secret key are required to mount a Native CephFS manila share. With a secret key, a manila user can impersonate a pre-existing ceph user and gain capabilities to manipulate resources that the manila user was never intended to have access to. It is possible to even obtain the default ceph "admin" user's key in this manner, and execute any commands as the ceph administrator. ### Affected Services / Software ### - - OpenStack Shared File Systems Service (Manila) versions Mitaka (2.0.0) through Victoria (11.0.0) - - Ceph Luminous (<=v12.2.13), Mimic (<=v13.2.10), Nautilus (<=v14.2.15), Octopus (<=v15.2.7) ### Discussion ### OpenStack Manila can provide users with Native CephFS shared file systems. When a user creates a "share" (short for "shared file system") via Manila, a CephFS "subvolume" is created on the Ceph cluster and exported. After creating their share, a user can specify who can have access to the share with the help of "cephx" client user names. A cephx client corresponds to Ceph Client Users [2]. When access is provided, a client user "access key" is returned via manila. A ceph client user account is required to access any ceph resource. This includes interacting with Ceph cluster infrastructure daemons (ceph-mgr, ceph-mds, ceph-mon, ceph-osd) or consuming Ceph storage via RBD, RGW or CephFS. Deployment and orchestration services like ceph-ansible, nfs-ganesha, kolla, tripleo need ceph client users to work, as do OpenStack services such as cinder, manila, glance and nova for their own interactions with Ceph. For the purpose of illustrating this vulnerability, we'll call them "pre-existing" users of the Ceph cluster. Another example of a pre-existing user includes the "admin" user that is created by default on the ceph cluster. In theory, manila's cephx users are no different from a ceph client user. When a manila user requests access to a share, a corresponding ceph user account is created if one already does not exist. If a ceph user account already exists, the existing capabilities of that user are adjusted to provide them permissions to access the manila share in question. There is no reasonable way for this mechanism to know what pre-existing ceph client users must be protected against unauthorized abuse. Therefore there is a risk that a manila user can claim to be a pre-existing ceph user to steal their access secret key. To resolve this issue, the ceph interface that manila uses was patched to no longer allow manila to claim a pre-existing user account that didn't create. By consequence this means that manila users cannot use cephx usernames that correspond to ceph client users that exist outside of manila. ### Recommended Actions ### #. Upgrade your ceph software to the latest patched releases of ceph to take advantage of the fix to this vulnerability. #. Audit cephx access keys provisioned via manila. You may use "ceph auth ls" and ensure that no clients have been compromised. If they have been, you may need to delete and recreate the client credentials to prevent unauthorized access. #. The audit can also be performed on manila by enumerating all CephFS shares and their access rules as a system administrator. If a reserved ceph client username has been used, you may deny access and recreate the client credential on ceph to refresh the access secret. No code changes were necessary in the OpenStack Shared File System service (manila). With an upgraded ceph, when manila users attempt to provide share access to a cephx username that they cannot use, the access rule's "state" attribute is set to "error" because this operation is no longer permitted. ### Patches ### The Ceph community has provided the following patches: Ceph Octopus: https://github.com/ceph/ceph/commit/1b8a634fdcd94dfb3ba650793fb1b6d09af65e05 Ceph Nautilus: https://github.com/ceph/ceph/commit/7e3e4e73783a98bb07ab399438eb3aab41a6fc8b Ceph Luminous: https://github.com/ceph/ceph/commit/956ceb853a58f6b6847b31fac34f2f0228a70579 The fixes are in the latest releases of Ceph Nautilus (14.2.16) and Ceph Octopus (15.2.8). The patch for Luminous was provided as a courtesy to possible users of OpenStack Manila, however the Ceph community no longer produces releases for Luminous or Mimic as they are end of life. See `here for information about ceph releases. <https://docs.ceph.com/en/latest/releases/general/>`_ ### Contacts / References ### Author: - - Pacha Ravi, Goutham gouthamr(a)redhat.com (Red Hat) Credits: - - Garbutt, John john(a)johngarbutt.com (StackHPC) - - Babel, Jahson jahson.babel(a)cc.in2p3.fr (Centre de Calcul de l'IN2P3) This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0087 Original LaunchPad Bug : https://launchpad.net/bugs/1904015 Mailing List : [Security] tag on openstack-discuss(a)lists.openstack.org OpenStack Security Project : https://launchpad.net/~openstack-ossg CVE: CVE-2020-27781 -----BEGIN PGP SIGNATURE----- wsFcBAEBCAAGBQJf2rsUAAoJENJP32eYWZR3gswP/iF9Q+itZjSbuUSaYuZn lKU2eu6AWJDW5AIarbO5i3ck6uWEzFCOPsVySpOaHvwUXZyYLd4TXqWIIszi hj0Q4Kjyl3kehg2uW8Krj3wnf5y2eCpCwzJFIjB9tClldcCXu2UviFm2dHDL kjDXj55USPSgZuOKKBIx8KXJCloA2lMpg0a9zM5LJA63ymqPycjeDkskxaTq seG2C6JyPdiQEtOFnl5DnflTA51jTEbxMwxwSMlkMlxCJGZyYf+2AqJ3MZkN G13kBWzwmUzNbmsY7QriOnl/jYwpMoHW01TZ8ga5O5ursV+9Lh0aZEfN7aMm KzEBPQOaT5vpeYG7yirA1kSAAebewkoABaZsExQ/IyzClF+aHbC+fr4k0NvK PBR0T8cwwQhzvcnNtyRe9mhdIQo/DIK+rqPLAeIlDjmzO6T2IzBC7OUDPlmr bszGSFxJ4ZOemJgOLUjcazxEX/k2Jj559cc/B8/Ak489f/xUoJYs0MmioF8a Ft9iA97ZvtoblfdCSuzVIl7r4jQL59qBfLykotU22ydwOT3VmILPVZRbPc+0 p+Nkj0xpGw3MEKtZr1+2OJpAsYhm+k6FoRvOd1wvHIB8QJXnCTjOZe8F/cLE axCIChFquoAN+YjwpUEVbo79sJv6wxEc1dmB5UT/InyELaDz3US7OIXWm0f6 UNdI =n0oA -----END PGP SIGNATURE-----

1 0

block.db Permission denied
by Seena Fallah 16 Dec '20

16 Dec '20

Hi. When I deployed an OSD with a separate db block it gets me a Permission denied on its path! I don't have any idea why but the only change I've done with my previous deployments was I change the osd_crush_initial_weight from 0 to 1. when I restart the host OSD can get up without any errors. I have tested another deployment with osd_crush_initial_weight=0 and it deployed successfully and again when I deploy with osd_crush_initial_weight=1 for another new node it gives me Permission denied! Here is the full trace: https://paste.ubuntu.com/p/6HDzwSrK3p/

1 0

Possibly unused client
by Alexander E. Patrakov 16 Dec '20

16 Dec '20

Hello, While working with a customer, I went through the output of "ceph auth list", and found a client entry that nobody can tell what it is used for. There is a strong suspicion that it is an unused left-over from old times, but again, nobody is sure. How can I confirm that it was not used for, say, the past week? Or, what logs should I turn on so that if it is used during the next week, it is mentioned there? -- Alexander E. Patrakov CV: http://pc.cd/PLz7

2 2

ceph stuck removing image from trash
by Andre Gebers 16 Dec '20

16 Dec '20

Hi, I'm running a 15.2.4 test cluster in a rook-ceph environment. The cluster is reporting HEALTH_OK but it seems it is stuck removing an image. Last section of 'ceph status' output: progress: Removing image replicapool/43def5e07bf47 from trash (6h) [............................] (remaining: 32y) This is now going for a couple of weeks and I was wondering if there is a way to speed it up? The cluster doesn't seem to be doing much judging from the system load. I've created this largish image to test what is possible with the setup but how do I get it out of the trash now? # rbd info --image-id 43def5e07bf47 -p replicapool rbd image 'csi-vol-cfaa1b00-1711-11eb-b9c9-2aa51e1e24e5': size 1 EiB in 274877906944 objects order 22 (4 MiB objects) snapshot_count: 0 id: 43def5e07bf47 block_name_prefix: rbd_data.43def5e07bf47 format: 2 features: layering op_features: flags: create_timestamp: Sun Oct 25 22:31:23 2020 access_timestamp: Sun Oct 25 22:31:23 2020 modify_timestamp: Sun Oct 25 22:31:23 2020 Any pointers how to resolve this issue are much appreciated. Regards Andre

4 3

issue on adding SSD to SATA cluster for db/wal
by Zhenshi Zhou 16 Dec '20

16 Dec '20

Hi all, I have a 14.2.15 cluster with all SATA OSDs. Now we plan to add SSDs in the cluster for db/wal usage. I checked the docs and found a command 'ceph-bluestore-tool' can deal with the issue. I added db/wal to the osd in my test environment but in the end it still get the warning message. "osd.0 spilled over 64 KiB metadata from 'db' device (7 MiB used of 8.0 GiB) to slow device" my procedure : sdd is the new disk for db/wal. sgdisk --new=1:0:+8GB --change-name=1:bluestore_block_db_0 --partition-guid=1:$(uuidgen) --mbrtogpt -- /dev/sdd sgdisk --new=2:0:+1GB --change-name=2:bluestore_block_wal_0 --partition-guid=2:$(uuidgen) --mbrtogpt -- /dev/sdd systemctl stop ceph-osd@0 CEPH_ARGS="--bluestore-block-db-size 8589934592" ceph-bluestore-tool bluefs-bdev-new-db --path /var/lib/ceph/osd/ceph-0 --dev-target /dev/sdd1 ceph-bluestore-tool bluefs-bdev-new-wal --path /var/lib/ceph/osd/ceph-0/ --dev-target /dev/sdd2 ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-0/ systemctl start ceph-osd@0 ceph tell osd.0 compact The warning message tells that there is some metadata still in the slow device. How can I deal with this issue? Thanks

2 2

Whether read I/O is accpted when the number of replica is under pool's min_size
by Satoru Takeuchi 16 Dec '20

16 Dec '20

Hi, Could you tell me whether read I/O is acxepted when the number of replicas is under pool's min_size? I read the official document and found that there is a difference of the effect of pool's min_size between pool's document and the pool's configuration document. Pool's document: https://docs.ceph.com/en/latest/rados/operations/pools/ > To set a minimum number of required replicas for I/O, you should use the min_size setting. Pool's configuration document. https://docs.ceph.com/en/latest/rados/configuration/pool-pg-config-ref/ > osd pool default min size Description Sets the minimum number of written replicas for objects in the pool in order to acknowledge a write operation to the client. The former says that min_size is related to both read/write I/O. On the other hands, the latter says it's only about write I/O. So which is correct description? As a result of my verification, the former looks correct. I created a RGW object in a pool having 3 replicas and its min_size is 2. Both read and write aren't accepted when 2 replicas are gone. Thanks, Satoru

2 2

multiple OSD crash, unfound objects
by Michael Thomas 15 Dec '20

15 Dec '20

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects. The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem). I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg…. I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects. I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below). The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours. My questions are: * Is there a way to force the cluster to query the possible locations sooner? * Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes? --Mike ceph -s: cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops. services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs task status: scrub status: mds.ceph4: idle data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................]

4 34

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

Weird ceph df
by Szabo, Istvan (Agoda) 15 Dec '20

15 Dec '20

Hi, It is a nautilus 14.2.13 ceph. The quota on the pool is 745GiB, how can be the stored data 788GiB? (2 replicas pool). Based on the used column it means just 334GiB is used because the pool has 2 replicas only. I don't understand. POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR k8s-dbss-w-mdc 12 788 GiB 202.42k 668 GiB 0.75 43 TiB N/A 745 GiB 202.42k 0 B 0 B Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"
by Hoan Nguyen Van 15 Dec '20

15 Dec '20

I found a merge request, ceph mon has new optíon :mon_sync_max_payload_keys https://github.com/ceph/ceph/commit/d6037b7f484e13cfc9136e63e4cf7fac6ad6896… My value of options mon_sync_max_payload_size is 4096. If mon_sync_max_payload_keys options is default, ceph mon may be not sync because slow opt. What do you think?

1 0

Slow Replication on Campus
by Vikas Rana 14 Dec '20

14 Dec '20

Hi Friends, We have 2 Ceph clusters on campus and we setup the second cluster as the DR solution. The images on the DR side are always behind the master. Ceph Version : 12.2.11 VMWARE_LUN0: global_id: 23460954-6986-4961-9579-0f2a1e58e2b2 state: up+replaying description: replaying, master_position=[object_number=2632711, tag_tid=24, entry_tid=1967382595], mirror_position=[object_number=1452837, tag_tid=24, entry_tid=456440697], entries_behind_master=1510941898 last_update: 2020-11-30 14:13:38 VMWARE_LUN1: global_id: cb579579-13b0-4522-b65f-c64ec44cbfaf state: up+replaying description: replaying, master_position=[object_number=1883943, tag_tid=28, entry_tid=1028822927], mirror_position=[object_number=1359161, tag_tid=28, entry_tid=358296085], entries_behind_master=670526842 last_update: 2020-11-30 14:13:33 Any suggestion on tuning or any parameters we can set on RBD-mirror to speed up the replication. Both cluster have very little activity. Appreciate your help. Thanks, -Vikas

2 1

iscsi and iser
by Marc Boisis 14 Dec '20

14 Dec '20

Hi, I would like to know if you support iser in gwcli like the traditional targetcli or if this is planned in a future version of ceph ? Thanks Marc

2 1

Ceph 15.2.4 segfault, msgr-worker
by Ivan Kurnosov 14 Dec '20

14 Dec '20

Hi Team, this night I have caught the following segfault. Nothing else looks suspicious (but I'm a quite newbie in ceph management, so perhaps just don't know where to look at). I could not google any similar segfault from anybody else. Was it a known problem fixed in later versions? The cluster has been running for quite a while now (several months) and this has happened for the first time. ``` debug 0> 2020-12-02T10:31:09.295+0000 7f3e61943700 -1 *** Caught signal (Segmentation fault) ** in thread 7f3e61943700 thread_name:msgr-worker-1 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) 1: (()+0x12dd0) [0x7f3e65933dd0] 2: (std::_Rb_tree<boost::intrusive_ptr<AsyncConnection>, boost::intrusive_ptr<AsyncConnection>, std::_Identity<boost::intrusive_ptr<AsyncConnection> >, std::less<boost::intrusive_ptr<AsyncConnection> >, std::allocator<boost::intrusive_ptr<AsyncConnection> > >::find(boost::intrusive_ptr<AsyncConnection> const&) const+0x2c) [0x55cee407ca7c] 3: (AsyncConnection::_stop()+0xab) [0x55cee40767eb] 4: (ProtocolV2::stop()+0x8f) [0x55cee40a189f] 5: (ProtocolV2::_fault()+0x133) [0x55cee40a1b03] 6: (ProtocolV2::handle_read_frame_preamble_main(std::unique_ptr<ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer>&&, int)+0x551) [0x55cee40a63d1] 7: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x3c) [0x55cee40a273c] 8: (AsyncConnection::process()+0x8a9) [0x55cee4079ab9] 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xcb7) [0x55cee3eceb67] 10: (()+0xdb914c) [0x55cee3ed414c] 11: (()+0xc2b73) [0x7f3e64f83b73] 12: (()+0x82de) [0x7f3e659292de] 13: (clone()+0x43) [0x7f3e64660e83] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_rwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7f3e3d88b700 / osd_srv_heartbt 7f3e41893700 / tp_osd_tp 7f3e4589b700 / tp_osd_tp 7f3e4f8af700 / rocksdb:dump_st 7f3e52a8c700 / safe_timer 7f3e53a8e700 / ms_dispatch 7f3e566b7700 / bstore_mempool 7f3e5d0ca700 / safe_timer 7f3e61943700 / msgr-worker-1 7f3e62144700 / msgr-worker-0 max_recent 10000 max_new 1000 log_file /var/lib/ceph/crash/2020-12-02T10:31:09.301492Z_84e8430f-30fd-469f-8e22-c2e1ccc675da/log --- end dump of recent events --- reraise_fatal: default handler for signal 11 didn't terminate the process? ``` -- With best regards, Ivan Kurnosov

2 1

Removing an applied service set
by Michael Wodniok 14 Dec '20

14 Dec '20

Hi, we created multiple CephFS, this invloved deploying mutliple mds-services using `ceph orch apply mds [...]`. Worked like a charm. Now the filesystem has been removed and the leftovers of the filesystem should also be removed, but I can't delete the services as cephadm/orchestration module is recreating them. What is the "official" way to delete this applied service set? Setting placement size to 0 is not possible in Ceph 15. Kind regards, Michael

2 2

Re: pool nearfull, 300GB rbd image occupies 11TB!
by mk 13 Dec '20

13 Dec '20

In fact journaling was enabled, is it enough to disable feature and pool shrinks automatically again? Or still any additional actions are required? — Max > On 13. Dec 2020, at 15:53, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > rbd status > rbd info > > If the ‘journaling’ flag is enabled, use ‘rbd feature’ to remove it from the image. > > > >> On Dec 13, 2020, at 6:22 AM, mk <mk(a)pop.de> wrote: >> >> Yes, few months ago I had enabled mirroring for few weeks and disabled again. Is there any additional actions has to be taken regarding journalling also ?? >> fyi. I also have copied rbd image into one newly created pool but after few weeks new pool grew up again to 11TB which is current state >> -- >> BR >> Max >> >>> On 13. Dec 2020, at 14:34, Jason Dillaman <jdillama(a)redhat.com> wrote: >>> >>>> On Sun, Dec 13, 2020 at 6:03 AM mk <mk(a)pop.de> wrote: >>>> >>>> rados ls -p ssdshop >>>> outputs 20MB of lines without any bench prefix >>>> ... >>>> rbd_data.d4993cc3c89825.00000000000074ec >>>> rbd_data.d4993cc3c89825.0000000000001634 >>>> journal_data.83.d4993cc3c89825.333485 >>>> journal_data.83.d4993cc3c89825.380648 >>>> journal_data.83.d4993cc3c89825.503838 >>>> ... >>> >>> If you have journal objects indexed at 333485 and 503838, that's >>> nearly 2TiBs of data right there. Sounds like you enabled journaling >>> and mirroring but perhaps never turned it off when you stopped using >>> it? >>> >>>>> On 13. Dec 2020, at 11:05, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: >>>>> >>>>> Any chance you might have orphaned `rados bench` objects ? This happens more than one might think. >>>>> >>>>> `rados ls > /tmp/out` >>>>> >>>>> Inspect the result. You should see a few administrative objects, some header and data objects for the RBD volume. If you see a zillion with names like `bench*` there’s your culprit. Those can be cleaned up. >>>>> >>>>> >>>>> >>>>>> On Dec 12, 2020, at 11:42 PM, mk <mk(a)pop.de> wrote: >>>>>> >>>>>> Hi folks, >>>>>> my cluster shows strange behavior, the only ssd pool on cluster with repsize 3 and pg/pgp size 512 >>>>>> which contains 300GB rbd image and only one snapshot occupies 11TB space! >>>>>> >>>>>> I have tried objectmap check / rebuild, fstrim etc. which couldn’t solve that problem, any help would be appreciated. >>>>>> >>>>>> >>>>>> ceph version 14.2.7 nautilus (stable) >>>>>> >>>>>> >>>>>> ceph df >>>>>> ------- >>>>>> RAW STORAGE: >>>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED >>>>>> hdd 107 TiB 68 TiB 39 TiB 39 TiB 36.45 >>>>>> ssd 21 TiB 11 TiB 11 TiB 11 TiB 50.78 >>>>>> TOTAL 128 TiB 78 TiB 50 TiB 50 TiB 38.84 >>>>>> >>>>>> POOLS: >>>>>> POOL ID STORED OBJECTS USED %USED MAX AVAIL >>>>>> ssdshop 83 3.5 TiB 517.72k 11 TiB 96.70 124 GiB >>>>>> >>>>>> >>>>>> rados df >>>>>> -------- >>>>>> POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR >>>>>> ssdshop 11 TiB 537040 28316 1611120 0 0 0 11482773 15 GiB 44189589 854 GiB 0 B 0 B >>>>>> >>>>>> >>>>>> >>>>>> rbd du -p ssdshop >>>>>> ----------------- >>>>>> NAME PROVISIONED USED >>>>>> shp-de-300gb.rbd@snap_2020-12-12_20:30:00 300 GiB 289 GiB >>>>>> shp-de-300gb.rbd 300 GiB 109 GiB >>>>>> <TOTAL> 300 GiB 398 GiB >>>>>> >>>>>> >>>>>> crush_rule >>>>>> ----------- >>>>>> rule ssd { >>>>>> id 3 >>>>>> type replicated >>>>>> min_size 1 >>>>>> max_size 10 >>>>>> step take dc1 class ssd >>>>>> step chooseleaf firstn 2 type rack >>>>>> step emit >>>>>> step take dc2 class ssd >>>>>> step chooseleaf firstn -1 type rack >>>>>> step emit >>>>>> } >>>>>> >>>>>> BR >>>>>> Max >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> >>> -- >>> Jason >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

2 2

pool nearfull, 300GB rbd image occupies 11TB!
by mk 13 Dec '20

13 Dec '20

Hi folks, my cluster shows strange behavior, the only ssd pool on cluster with repsize 3 and pg/pgp size 512 which contains 300GB rbd image and only one snapshot occupies 11TB space! I have tried objectmap check / rebuild, fstrim etc. which couldn’t solve that problem, any help would be appreciated. ceph version 14.2.7 nautilus (stable) ceph df ------- RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 107 TiB 68 TiB 39 TiB 39 TiB 36.45 ssd 21 TiB 11 TiB 11 TiB 11 TiB 50.78 TOTAL 128 TiB 78 TiB 50 TiB 50 TiB 38.84 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL ssdshop 83 3.5 TiB 517.72k 11 TiB 96.70 124 GiB rados df -------- POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR ssdshop 11 TiB 537040 28316 1611120 0 0 0 11482773 15 GiB 44189589 854 GiB 0 B 0 B rbd du -p ssdshop ----------------- NAME PROVISIONED USED shp-de-300gb.rbd@snap_2020-12-12_20:30:00 300 GiB 289 GiB shp-de-300gb.rbd 300 GiB 109 GiB <TOTAL> 300 GiB 398 GiB crush_rule ----------- rule ssd { id 3 type replicated min_size 1 max_size 10 step take dc1 class ssd step chooseleaf firstn 2 type rack step emit step take dc2 class ssd step chooseleaf firstn -1 type rack step emit } BR Max

3 4

pool nearfull, 300GB rbd image occupies 11TB!
by mk 13 Dec '20

13 Dec '20

Hi folks, my cluster shows strange behavior, the only ssd pool on cluster with repsize 3 and pg/pgp size 512 which contains 300GB rbd image and only one snapshot occupies 11TB space! I have tried objectmap check / rebuild, fstrim etc. which couldn’t solve that problem, any help would be appreciated. ceph version 14.2.7 nautilus (stable) ceph df ------- RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 107 TiB 68 TiB 39 TiB 39 TiB 36.45 ssd 21 TiB 11 TiB 11 TiB 11 TiB 50.78 TOTAL 128 TiB 78 TiB 50 TiB 50 TiB 38.84 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL ssdshop 83 3.5 TiB 517.72k 11 TiB 96.70 124 GiB rados df -------- POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR ssdshop 11 TiB 537040 28316 1611120 0 0 0 11482773 15 GiB 44189589 854 GiB 0 B 0 B rbd du -p ssdshop ----------------- NAME PROVISIONED USED shp-de-300gb.rbd@snap_2020-12-12_20:30:00 300 GiB 289 GiB shp-de-300gb.rbd 300 GiB 109 GiB <TOTAL> 300 GiB 398 GiB crush_rule ----------- rule ssd { id 3 type replicated min_size 1 max_size 10 step take dc1 class ssd step chooseleaf firstn 2 type rack step emit step take dc2 class ssd step chooseleaf firstn -1 type rack step emit } BR Max

1 0

Third nautilus OSD dead in 11 days - FAILED is_valid_io(off, len)
by Jonas Jelten 12 Dec '20

12 Dec '20

Hi! Yesterday a third OSD died with a failed assertion, and it can no longer boot. It's the third OSD within 11 days. There's already a tracker issue: https://tracker.ceph.com/issues/48276 2020-12-11 20:06:51.839 7fe2b5ffd700 -1 /build/ceph-14.2.13/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::aio_write(uint64_t, ceph::bufferlist&, IOContext*, bool, int)' thread 7fe2b5ffd /build/ceph-14.2.13/src/os/bluestore/KernelDevice.cc: 864: FAILED ceph_assert(is_valid_io(off, len)) ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55ff9fac9eea] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55ff9faca0c5] 3: (KernelDevice::aio_write(unsigned long, ceph::buffer::v14_2_0::list&, IOContext*, bool, int)+0x19e9) [0x55ffa01272a9] 4: (BlueStore::_deferred_submit_unlock(BlueStore::OpSequencer*)+0x621) [0x55ff9ffe87f1] 5: (BlueStore::deferred_try_submit()+0x3e0) [0x55ff9ffe9da0] 6: (BlueStore::_kv_finalize_thread()+0x6c6) [0x55ffa002fcc6] 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x55ffa006066d] 8: (()+0x76db) [0x7fe2c61456db] 9: (clone()+0x3f) [0x7fe2c4ee571f] When the OSD tries to boot (now with 14.2.15), it fails again: -4> 2020-12-11 20:07:05.160 7ff2d15e7c00 1 bluestore(/var/lib/ceph/osd/ceph-147) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2 -3> 2020-12-11 20:07:05.160 7ff2d15e7c00 1 bluestore(/var/lib/ceph/osd/ceph-147) _upgrade_super from 2, latest 2 -2> 2020-12-11 20:07:05.160 7ff2d15e7c00 1 bluestore(/var/lib/ceph/osd/ceph-147) _upgrade_super done -1> 2020-12-11 20:07:05.184 7ff2d15e7c00 -1 /build/ceph-14.2.15/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::aio_write(uint64_t, ceph::bufferlist&, IOContext*, bool, int)' thread 7ff2d15e7c00 time 2020-12-11 20:07:05.186142 /build/ceph-14.2.15/src/os/bluestore/KernelDevice.cc: 888: FAILED ceph_assert(is_valid_io(off, len)) ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x564ad1323fba] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564ad1324195] 3: (KernelDevice::aio_write(unsigned long, ceph::buffer::v14_2_0::list&, IOContext*, bool, int)+0x16da) [0x564ad198154a] 4: (BlueStore::_deferred_submit_unlock(BlueStore::OpSequencer*)+0x621) [0x564ad1842911] 5: (BlueStore::deferred_try_submit()+0x3e0) [0x564ad1843ec0] 6: (BlueStore::_osr_drain_all()+0x1a5) [0x564ad1845255] 7: (BlueStore::_deferred_replay()+0x1c7) [0x564ad188b297] 8: (BlueStore::_mount(bool, bool)+0x83e) [0x564ad1892e1e] 9: (OSD::init()+0x3f3) [0x564ad13d3db3] 10: (main()+0x5214) [0x564ad132ccf4] 11: (__libc_start_main()+0xe7) [0x7ff2cde60bf7] 12: (_start()+0x2a) [0x564ad135e72a] Anyone else with these symptoms? Any ideas how we can track it down? -- Jonas

2 2