November 2020 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

fixing future rctimes
by Dan van der Ster 23 Mar '21

23 Mar '21

Hi all, We have a few subdirs with an rctime in the future. # getfattr -n ceph.dir.rctime session # file: session ceph.dir.rctime="2576387188.090" I can't find any subdir or item in that directory with that rctime, so I presume that there was previously a file and that rctime cannot go backwards [1] Is there any way to fix these rctimes so they show the latest ctime of the subtree? Also -- are we still relying on the client clock to set the rctime / ctime of a file? Would it make sense to limit ctime/rctime for any update to the current time on the MDS ? Best Regards, Dan [1] https://github.com/ceph/ceph/pull/24023/commits/920ef964311a61fcc6c0d6671b7…

2 3

Remapped PGs
by David Orman 02 Mar '21

02 Mar '21

Hi, We see that we have 5 'remapped' PGs, but are unclear why/what to do about it. We shifted some target ratios for the autobalancer and it resulted in this state. When adjusting ratio, we noticed two OSDs go down, but we just restarted the container for those OSDs with podman, and they came back up. Here's status output: ################### root@ceph01:~# ceph status INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w) mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl, ceph02.gcglcg, ceph04.smbdew, ceph05.yropto osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs data: pools: 3 pools, 1057 pgs objects: 18.00M objects, 69 TiB usage: 119 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 1056 active+clean 1 active+clean+scrubbing+deep io: client: 859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr root@ceph01:~# ################### When I look at ceph pg dump, I don't see any marked as remapped: ################### root@ceph01:~# ceph pg dump |grep remapped INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 dumped all root@ceph01:~# ################### Any idea what might be going on/how to recover? All OSDs are up. Health is 'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman 2.0.3.

2 4

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

Default data pool in CEPH
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi, I have a CEPH 15.2.4 running in a docker. How to configure for use a specific data pool? i try put the follow line in the ceph.conf but the changes not working. . [client.myclient] rbd default data pool = Mydatapool I need it to configure for erasure pool with cloudstack Can help me? , where is the ceph conf we i need configure? Thanks. -- Untitled Document

4 6

Re: NFS Ganesha NFSv3
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi Thanks for the reply. cephadm runs ceph containers automatically. How to set privileged mode in ceph container? -- > El 23/9/20 a las 13:24, Daniel Gryniewicz escribió: >> NFSv3 needs privileges to connect to the portmapper. Try running >> your docker container in privileged mode, and see if that helps. >> >> Daniel >> >> On 9/23/20 11:42 AM, Gabriel Medve wrote: >>> Hi, >>> >>> I have a CEPH 15.2.5 running in a docker , i configure nfs ganesha >>> with nfs version 3 but i can not mount it. >>> If configure ganesha with nfs version 4 i can mounted without >>> problems but i need the version 3 . >>> >>> The error is mount.nfs: Protocol not supported >>> >>> Can help me? >>> >>> Thanks. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- > Untitled Document

2 1

Possible to disable check: x pool(s) have no replicas configured
by Marc Roos 23 Feb '21

23 Feb '21

Is it possible to disable checking on 'x pool(s) have no replicas configured', so I don't have this HEALTH_WARN constantly. Or is there some other disadvantage of keeping some empty 1x replication test pools?

2 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Planning: Ceph User Survey 2020
by Mike Perez 28 Jan '21

28 Jan '21

Hi everyone, The Ceph User Survey 2020 is being planned by our working group. Please review the draft survey pdf, and let's discuss any changes. You may also join us in the next meeting *on November 25th at 12pm *PT https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group https://tracker.ceph.com/attachments/download/5260/Ceph%20User%20Survey%202… We're aiming to have something ready by mid-December. -- Mike Perez he/him Ceph / Rook / RDO / Gluster Community Architect Open-Source Program Office (OSPO) M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

5 6

Ceph 15.2.3 on Ubuntu 20.04 with odroid xu4 / python thread Problem
by Dominik H 16 Jan '21

16 Jan '21

Hi all, i try to run ceph client tools on an odroid xu4 (armhf) with Ubuntu 20.04 on python 3.8.5. Unfortunately there is the following error on each "ceph" command (even in ceph --help) Traceback (most recent call last): File "/usr/bin/ceph", line 1275, in <module> retval = main() File "/usr/bin/ceph", line 981, in main cluster_handle = run_in_thread(rados.Rados, File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1342, in run_in_thread raise Exception("timed out") Exception: timed out With this Server I access an existing Ceph-Cluster with the same hardware. I checked the code part, there is just a thread start and a join (waiting for finish a RadosThread). Maybe this is a python error in combination with armhf architecture? Maybe someone can help. Thanks and greetings Dominik

3 8

Bluestore migration: per-osd device copy
by Chris Dunlop 05 Jan '21

05 Jan '21

Hi, The docs have scant detail on doing a migration to bluestore using a per-osd device copy: https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-o… This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a copy function (all the way from v9 to current). Has anyone actually tried doing this? Is there any further detail available on what is involved, e.g. a broad outline of the steps? Of course, detailed instructions would be even better, even if accompanied by "here be dragons!" warnings. Cheers, Chris

3 6

diskprediction_local fails with python3-sklearn 0.22.2
by Eric Dold 22 Dec '20

22 Dec '20

Hello the mgr module diskprediction_local fails under ubuntu 20.04 focal with python3-sklearn version 0.22.2 Ceph version is 15.2.3 when the module is enabled i get the following error: File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in serve self.predict_all_devices() File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in predict_all_devices result = self._predict_life_expentancy(devInfo['devid']) File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in _predict_life_expentancy predicted_result = obj_predictor.predict(predict_datas) File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457, in predict pred = clf.predict(ordered_data) File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in predict if self.break_ties and self.decision_function_shape == 'ovo': AttributeError: 'SVC' object has no attribute 'break_ties' Best Regards Eric

2 1

osd_pglog memory hoarding - another case
by Kalle Happonen 22 Dec '20

22 Dec '20

Hello all, wrt: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7IMIWCKIHXN… Yesterday we hit a problem with osd_pglog memory, similar to the thread above. We have a 56 node object storage (S3+SWIFT) cluster with 25 OSD disk per node. We run 8+3 EC for the data pool (metadata is on replicated nvme pool). The cluster has been running fine, and (as relevant to the post) the memory usage has been stable at 100 GB / node. We've had the default pg_log of 3000. The user traffic doesn't seem to have been exceptional lately. Last Thursday we updated the OSDs from 14.2.8 -> 14.2.13. On Friday the memory usage on OSD nodes started to grow. On each node it grew steadily about 30 GB/day, until the servers started OOM killing OSD processes. After a lot of debugging we found that the pg_logs were huge. Each OSD process pg_log had grown to ~22GB, which we naturally didn't have memory for, and then the cluster was in an unstable situation. This is significantly more than the 1,5 GB in the post above. We do have ~20k pgs, which may directly affect the size. We've reduced the pg_log to 500, and started offline trimming it where we can, and also just waited. The pg_log size dropped to ~1,2 GB on at least some nodes, but we're still recovering, and have a lot of ODSs down and out still. We're unsure if version 14.2.13 triggered this, or if the osd restarts triggered this (or something unrelated we don't see). This mail is mostly to figure out if there are good guesses why the pg_log size per OSD process exploded? Any technical (and moral) support is appreciated. Also, currently we're not sure if 14.2.13 triggered this, so this is also to put a data point out there for other debuggers. Cheers, Kalle Happonen

4 16

v14.2.15 Nautilus released
by David Galloway 18 Dec '20

18 Dec '20

This is the 15th backport release in the Nautilus series. This release fixes a ceph-volume regression introduced in v14.2.13 and includes few other fixes. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-15-nautilus-released Notable Changes --------------- * ceph-volume: Fixes lvm batch --auto, which breaks backward compatibility when using non rotational devices only (SSD and/or NVMe). * BlueStore: Fixes a bug in collection_list_legacy which makes pgs inconsistent during scrub when running mixed versions of osds, prior to 14.2.12 with newer. * MGR: progress module can now be turned on/off, using the commands: `ceph progress on` and `ceph progress off`. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.15.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: afdd217ae5fb1ed3f60e16bd62357ca58cc650e5

4 4

v15.2.4 Octopus released
by David Galloway 17 Dec '20

17 Dec '20

We're happy to announce the fourth bugfix release in the Octopus series. In addition to a security fix in RGW, this release brings a range of fixes across all components. We recommend that all Octopus users upgrade to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-4-octopus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * Cephadm: There were a lot of small usability improvements and bug fixes: * Grafana when deployed by Cephadm now binds to all network interfaces. * `cephadm check-host` now prints all detected problems at once. * Cephadm now calls `ceph dashboard set-grafana-api-ssl-verify false` when generating an SSL certificate for Grafana. * The Alertmanager is now correctly pointed to the Ceph Dashboard * `cephadm adopt` now supports adopting an Alertmanager * `ceph orch ps` now supports filtering by service name * `ceph orch host ls` now marks hosts as offline, if they are not accessible. * Cephadm can now deploy NFS Ganesha services. For example, to deploy NFS with a service id of mynfs, that will use the RADOS pool nfs-ganesha and namespace nfs-ns:: ceph orch apply nfs mynfs nfs-ganesha nfs-ns * Cephadm: `ceph orch ls --export` now returns all service specifications in yaml representation that is consumable by `ceph orch apply`. In addition, the commands `orch ps` and `orch ls` now support `--format yaml` and `--format json-pretty`. * Cephadm: `ceph orch apply osd` supports a `--preview` flag that prints a preview of the OSD specification before deploying OSDs. This makes it possible to verify that the specification is correct, before applying it. * RGW: The `radosgw-admin` sub-commands dealing with orphans -- `radosgw-admin orphans find`, `radosgw-admin orphans finish`, and `radosgw-admin orphans list-jobs` -- have been deprecated. They have not been actively maintained and they store intermediate results on the cluster, which could fill a nearly-full cluster. They have been replaced by a tool, currently considered experimental, `rgw-orphan-list`. * RBD: The name of the rbd pool object that is used to store rbd trash purge schedule is changed from "rbd_trash_trash_purge_schedule" to "rbd_trash_purge_schedule". Users that have already started using `rbd trash purge schedule` functionality and have per pool or namespace schedules configured should copy "rbd_trash_trash_purge_schedule" object to "rbd_trash_purge_schedule" before the upgrade and remove "rbd_trash_purge_schedule" using the following commands in every RBD pool and namespace where a trash purge schedule was previously configured:: rados -p <pool-name> [-N namespace] cp rbd_trash_trash_purge_schedule rbd_trash_purge_schedule rados -p <pool-name> [-N namespace] rm rbd_trash_trash_purge_schedule or use any other convenient way to restore the schedule after the upgrade. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7447c15c6ff58d7fce91843b705a268a1917325c -- David Galloway Systems Administrator, RDU Ceph Engineering IRC: dgalloway

4 5

multiple OSD crash, unfound objects
by Michael Thomas 15 Dec '20

15 Dec '20

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects. The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem). I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg…. I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects. I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below). The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours. My questions are: * Is there a way to force the cluster to query the possible locations sooner? * Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes? --Mike ceph -s: cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops. services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs task status: scrub status: mds.ceph4: idle data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................]

4 34

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

rgw index shard much larger than others
by Dan van der Ster 08 Dec '20

08 Dec '20

Dear friends, Running 14.2.11, we have one particularly large bucket with a very strange distribution of objects among the shards. The bucket has 512 shards, and most shards have ~75k entries, but shard 0 has 1.75M entries: # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.0 | wc -l 1752085 # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.1 | wc -l 78388 # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.2 | wc -l 78764 We had resharded this bucket (manually) from 32 up to 512 shards just before upgrading from 12.2.12 to 14.2.11 a couple weeks ago. Any idea why shard .0 is getting such an imbalance of entries? Should we manually reshard this bucket again? Thanks! Dan

3 6

block.db/block.wal device performance dropped after upgrade to 14.2.10
by Vladimir Prokofev 04 Dec '20

04 Dec '20

Good day, cephers! We've recently upgraded our cluster from 14.2.8 to 14.2.10 release, also performing full system packages upgrade(Ubuntu 18.04 LTS). After that performance significantly dropped, main reason beeing that journal SSDs are now have no merges, huge queues, and increased latency. There's a few screenshots in attachments. This is for an SSD journal that supports block.db/block.wal for 3 spinning OSDs, and it looks like this for all our SSD block.db/wal devices across all nodes. Any ideas what may cause that? Maybe I've missed something important in release notes?

6 20

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Many ceph commands hang. broken mgr?
by Paul Mezzanini 03 Dec '20

03 Dec '20

Ever since we jumped from 14.2.9 to .12 (and beyond) a lot of the ceph commands just hang. The mgr daemon also just stops responding to our Prometheus scrapes occasionally. A daemon restart and it wakes back up. I have nothing pointing to these being related but it feels that way. I also tried to get device health monitoring with smart up and running around that upgrade time. It never seemed to be able to pull in and report on the health across the drives. I did see the osd process firing off smartctl on occasion though so it was trying to do something. Again, I have nothing pointing to this being related but it feels like it may be. Some commands that currently hang: ceph osd pool autoscale-status ceph balancer * ceph iostat (oddly, this spit out a line of all 0 stats once and then hung) ceph fs status toggling ceph device monitoring on or off and a lot of the device health stuff too Mgr logs on disk show flavors of this: 2020-11-24 13:05:07.883 7f19e2c40700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{,",p,r,e,f,i,x,",:, ,",o,s,d, ,p,e,r,f,",,, ,",f,o,r,m,a,t,",:, ,",j,s,o,n,",}]: dispatch 2020-11-24 13:05:07.895 7f19e2c40700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{,",p,r,e,f,i,x,",:, ,",o,s,d, ,p,o,o,l, ,s,t,a,t,s,",,, ,",f,o,r,m,a,t,",:, ,",j,s,o,n,",}]: dispatch 2020-11-24 13:05:08.567 7f19e1c3e700 0 log_channel(cluster) log [DBG] : pgmap v587: 17149 pgs: 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 55 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 17082 active+clean; 2.1 PiB data, 3.5 PiB used, 2.9 PiB / 6.4 PiB avail; 108 MiB/s rd, 53 MiB/s wr, 1.20k op/s; 7525420/9900121381 objects misplaced (0.076%); 99 MiB/s, 40 objects/s recovering ceph status: cluster: id: 971a5242-f00d-421e-9bf4-5a716fcc843a health: HEALTH_WARN 1 nearfull osd(s) 1 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon-01,ceph-mon-03,ceph-mon-02 (age 4h) mgr: ceph-mon-01(active, since 97s), standbys: ceph-mon-03, ceph-mon-02 mds: cephfs:1 {0=ceph-mds-02=up:active} 3 up:standby osd: 843 osds: 843 up (since 13d), 843 in (since 2w); 10 remapped pgs rgw: 1 daemon active (ceph-rgw-01) task status: scrub status: mds.ceph-mds-02: idle data: pools: 16 pools, 17149 pgs objects: 1.61G objects, 2.1 PiB usage: 3.5 PiB used, 2.9 PiB / 6.4 PiB avail pgs: 6482000/9900825469 objects misplaced (0.065%) 17080 active+clean 54 active+clean+scrubbing+deep 9 active+remapped+backfilling 5 active+clean+scrubbing 1 active+remapped+backfill_wait io: client: 877 MiB/s rd, 1.8 GiB/s wr, 1.91k op/s rd, 3.33k op/s wr recovery: 136 MiB/s, 55 objects/s ceph config dump: WHO MASK LEVEL OPTION VALUE RO global advanced cluster_network 192.168.42.0/24 * global advanced mon_max_pg_per_osd 400 global advanced mon_pg_warn_max_object_skew -1.000000 global dev mon_warn_on_pool_pg_num_not_power_of_two false global advanced osd_max_backfills 2 global advanced osd_max_scrubs 4 global advanced osd_scrub_during_recovery false global advanced public_network 1xx.xx.171.0/24 10.16.171.0/24 * mon advanced mon_allow_pool_delete true mgr advanced mgr/balancer/mode none mgr advanced mgr/devicehealth/enable_monitoring false osd advanced bluestore_compression_mode passive osd advanced osd_deep_scrub_large_omap_object_key_threshold 2000000 osd advanced osd_op_queue_cut_off high * osd advanced osd_scrub_load_threshold 5.000000 mds advanced mds_beacon_grace 300.000000 mds basic mds_cache_memory_limit 16384000000 mds advanced mds_log_max_segments 256 client advanced rbd_default_features 5 client.libvirt advanced admin_socket /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok * client.libvirt basic log_file /var/log/ceph/qemu-guest-$pid.log * /etc/ceph/ceph.conf is the stub file with fsid and the mons listed. Yes I have a drive that just started to tickle the full warn limit. That's what pulled me back into the "I should fix this" mode. I'm manually adjusting the weight on that one for the time being along with slowly lowering pg_num on an oversized pool. The cluster still has this issue when in health_ok. I'm free to do a lot of debugging and poking around even though this is our production cluster. The only service I refuse to play around with is the MDS. That one bites back. Does anyone have more ideas on where to look to try and figure out what's going on? -- Paul Mezzanini Sr Systems Administrator / Engineer, Research Computing Information & Technology Services Finance & Administration Rochester Institute of Technology o:(585) 475-3245 | pfmeec(a)rit.edu CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information. ------------------------

2 7

slow down keys/s in recovery
by Seena Fallah 03 Dec '20

03 Dec '20

Hi all, Is there any configuration to slow down keys/s in recovery mode? Thanks.

4 17

How to create single OSD with SSD db device with cephadm
by 胡玮文 03 Dec '20

03 Dec '20

Hi all, I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a single new OSD daemon with a db device on SSD. But I can’t find any documentation about this. I have tried: 1. Using web dashboard. This requires at least one filter to proceed (type, vendor, model or size). But I just want to select the block device manually. 2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based. 3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db device in this way. 4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’ does. 5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I can’t figure out what is that. Any help is appreciated. Thanks.

2 4

replace osd with Octopus
by Tony Liu 02 Dec '20

02 Dec '20

Hi, I did some search about replacing osd, and found some different steps, probably for different release? Is there recommended process to replace an osd with Octopus? Two cases here: 1) replace HDD whose WAL and DB are on a SSD. 1-1) failed disk is replaced by the same model. 1-2) working disk is replaced by bigger one. 2) replace the SSD holding WAL and DB for multiple HDDs. Thanks! Tony

4 13

osd out cant' bring it back online
by Oliver Weinmann 01 Dec '20

01 Dec '20

Hi, I'm still evaluating ceph 15.2.5 in a lab so the problem is not really hurting me, but I want to understand it and hopefully fix it. It is a good practice. To test the resilience of the cluster I try to break it by doing all kinds of things. Today I powered off (clean shutdown) one osd node and powered it back on. Last time I tried this there was no problem getting it back online. After a few minutes the cluster health was back to ok. This time it stayed degraded forever. I checked and noticed that the service osd.0 on the osd node was failing. So i used google and there people recommended to simply delete the osd and re-create it. I tried it and still can't get the osd back in service. First I removed the osd: [root@gedasvl02 ~]# ceph osd out 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 osd.0 is already out. [root@gedasvl02 ~]# ceph auth del 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 Error EINVAL: bad entity name [root@gedasvl02 ~]# ceph auth del osd.0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 updated [root@gedasvl02 ~]# ceph osd rm 0 INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 removed osd.0 [root@gedasvl02 ~]# ceph osd tree INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.43658 root default -7 0.21829 host gedaopl01 2 ssd 0.21829 osd.2 up 1.00000 1.00000 -3 0 host gedaopl02 -5 0.21829 host gedaopl03 3 ssd 0.21829 osd.3 up 1.00000 1.00000 Looks ok it's gone... Then i zapped it: [root@gedasvl02 ~]# ceph orch device zap gedaopl02 /dev/sdb --force INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 INFO:cephadm:/usr/bin/podman:stderr WARNING: The same type, major and minor should not be used for multiple devices. INFO:cephadm:/usr/bin/podman:stderr --> Zapping: /dev/sdb INFO:cephadm:/usr/bin/podman:stderr --> Zapping lvm member /dev/sdb. lv_path is /dev/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a/osd-block-3a79800d-2a19-45d8-a850-82c6a8113323 INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a/osd-block-3a79800d-2a19-45d8-a850-82c6a8113323 bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/podman:stderr 10+0 records out INFO:cephadm:/usr/bin/podman:stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.0314447 s, 333 MB/s INFO:cephadm:/usr/bin/podman:stderr stderr: INFO:cephadm:/usr/bin/podman:stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-3bf1bb28-0858-4464-a848-d7f56319b40a INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/sbin/vgremove -v -f ceph-3bf1bb28-0858-4464-a848-d7f56319b40a INFO:cephadm:/usr/bin/podman:stderr stderr: Removing ceph--3bf1bb28--0858--4464--a848--d7f56319b40a-osd--block--3a79800d--2a19--45d8--a850--82c6a8113323 (253:0) INFO:cephadm:/usr/bin/podman:stderr stderr: Archiving volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" metadata (seqno 5). INFO:cephadm:/usr/bin/podman:stderr stderr: Releasing logical volume "osd-block-3a79800d-2a19-45d8-a850-82c6a8113323" INFO:cephadm:/usr/bin/podman:stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" (seqno 6). INFO:cephadm:/usr/bin/podman:stderr stdout: Logical volume "osd-block-3a79800d-2a19-45d8-a850-82c6a8113323" successfully removed INFO:cephadm:/usr/bin/podman:stderr stderr: Removing physical volume "/dev/sdb" from volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" INFO:cephadm:/usr/bin/podman:stderr stdout: Volume group "ceph-3bf1bb28-0858-4464-a848-d7f56319b40a" successfully removed INFO:cephadm:/usr/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/podman:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/podman:stderr 10+0 records out INFO:cephadm:/usr/bin/podman:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0355641 s, 295 MB/s INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: <Raw Device: /dev/sdb> And re-added it: [root@gedasvl02 ~]# ceph orch daemon add osd gedaopl02:/dev/sdb INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 Created osd(s) 0 on host 'gedaopl02' But the osd is still out... [root@gedasvl02 ~]# ceph osd tree INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.43658 root default -7 0.21829 host gedaopl01 2 ssd 0.21829 osd.2 up 1.00000 1.00000 -3 0 host gedaopl02 -5 0.21829 host gedaopl03 3 ssd 0.21829 osd.3 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000 Looking at the cluster log in the webui i see the following error: Failed to apply osd.dashboard-admin-1606745745154 spec DriveGroupSpec(name=dashboard-admin-1606745745154->placement=PlacementSpec(host_pattern='*'), service_id='dashboard-admin-1606745745154', service_type='osd', data_devices=DeviceSelection(size='223.6GB', rotational=False, all=False), osd_id_claims={}, unmanaged=False, filter_logic='AND', preview_only=False): No filters applied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 2108, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/module.py", line 2005, in _apply_service self.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 43, in create_from_spec ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 127, in prepare_drivegroup drive_selection = DriveSelection(drive_group, inventory_for_host) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 32, in __init__ self._data = self.assign_devices(self.spec.data_devices) File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 138, in assign_devices if not all(m.compare(disk) for m in FilterGenerator(device_filter)): File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 138, in <genexpr> if not all(m.compare(disk) for m in FilterGenerator(device_filter)): File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/matchers.py", line 410, in compare raise Exception("No filters applied") Exception: No filters applied I have another error "pgs undersized", maybe this is also causing trouble? [root@gedasvl02 ~]# ceph -s INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: d0920c36-2368-11eb-a5de-005056b703af health: HEALTH_WARN Degraded data redundancy: 13142/39426 objects degraded (33.333%), 176 pgs degraded, 225 pgs undersized services: mon: 1 daemons, quorum gedasvl02 (age 2w) mgr: gedasvl02.vqswxg(active, since 2w), standbys: gedaopl02.yrwzqh mds: cephfs:1 {0=cephfs.gedaopl01.zjuhem=up:active} 1 up:standby osd: 3 osds: 2 up (since 4d), 2 in (since 94m) task status: scrub status: mds.cephfs.gedaopl01.zjuhem: idle data: pools: 7 pools, 225 pgs objects: 13.14k objects, 77 GiB usage: 148 GiB used, 299 GiB / 447 GiB avail pgs: 13142/39426 objects degraded (33.333%) 176 active+undersized+degraded 49 active+undersized io: client: 0 B/s rd, 6.1 KiB/s wr, 0 op/s rd, 0 op/s wr Best Regards, Oliver

2 5

librdbpy examples
by Vladimir Prokofev 30 Nov '20

30 Nov '20

Hi. Just want to notice that if you google for ceph python lib examples it leads to 404 https://www.google.ru/search?hl=ru&q=ceph+python+rbd https://docs.ceph.com/en/latest/rbd/api/librbdpy/ Some 3rd party sites and chinese version works fine though http://docs.ceph.org.cn/rbd/librbdpy/ https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/b…

1 0

high memory usage in osd_pglog
by Robert Brooks 30 Nov '20

30 Nov '20

We are seeing very high osd_pglog usage in mempools for ceph osds. For example... "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluestore_alloc_bytes": 41857200, "bluestore_alloc_items": 523215, "bluestore_cache_data_bytes": 50876416, "bluestore_cache_data_items": 1326, "bluestore_cache_onode_bytes": 6814080, "bluestore_cache_onode_items": 13104, "bluestore_cache_other_bytes": 57793850, "bluestore_cache_other_items": 2599669, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_txc_bytes": 29904, "bluestore_txc_items": 42, "bluestore_writing_deferred_bytes": 733191, "bluestore_writing_deferred_items": 96, "bluestore_writing_bytes": 0, "bluestore_writing_items": 0, "bluefs_bytes": 101400, "bluefs_items": 1885, "buffer_anon_bytes": 21505818, "buffer_anon_items": 14949, "buffer_meta_bytes": 1161512, "buffer_meta_items": 13199, "osd_bytes": 1962920, "osd_items": 167, "osd_mapbl_bytes": 825079, "osd_mapbl_items": 17, "osd_pglog_bytes": 14099381936, "osd_pglog_items": 134285429, "osdmap_bytes": 734616, "osdmap_items": 26508, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "mds_co_bytes": 0, "mds_co_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, Where roughly 14g is required for pg_logs. Cluster has 106 OSD and 2432 placement groups. The pg log count for placement groups is much less than 134285429 logs. Top counts are... 1486 1.41c 883 7.3 834 7.f 683 7.13 669 7.a 623 7.5 565 7.8 560 7.1c 546 7.16 544 7.19 Summing these gives 21594 pg logs. Overall the performance of the cluster is poor, OSD memory usage is high (20-30G resident), and with a moderate workload we are seeing iowait on OSD hosts. The memory allocated to caches appears to be low, I believe because osd_pglog is taking most of the available memory. Regards, Rob -- ******************************************************************* This message was sent from RiskIQ, and is intended only for the designated recipient(s). It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not a designated recipient, you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you. *******************************************************************

2 2

CEPH-ISCSI fails when restarting rbd-target-api and won't work anymore
by Hamidreza Hosseini 30 Nov '20

30 Nov '20

I have an issue on ceph-iscsi ( ubuntu 20 LTS and Ceph 15.2.6) after I restart rbd-target-api, it fails and not starting again: ``` sudo systemctl status rbd-target-api.service ● rbd-target-api.service - Ceph iscsi target configuration API Loaded: loaded (/lib/systemd/system/rbd-target-api.service; enabled; vendor preset: enabled) Active: deactivating (stop-sigterm) since Sat 2020-11-28 17:01:40 +0330; 20s ago Main PID: 37651 (rbd-target-api) Tasks: 55 (limit: 9451) Memory: 141.4M CGroup: /system.slice/rbd-target-api.service ├─15289 /usr/bin/python3 /usr/bin/rbd-target-api └─37651 /usr/bin/python3 /usr/bin/rbd-target-api Nov 28 14:36:53 dev11 systemd[1]: Started Ceph iscsi target configuration API. Nov 28 14:36:54 dev11 rbd-target-api[37651]: Started the configuration object watcher Nov 28 14:36:54 dev11 rbd-target-api[37651]: Processing osd blacklist entries for this node Nov 28 14:36:54 dev11 rbd-target-api[37651]: Checking for config object changes every 1s Nov 28 14:36:55 dev11 rbd-target-api[37651]: Reading the configuration object to update local LIO configuration Nov 28 14:36:55 dev11 rbd-target-api[37651]: Processing Gateway configuration Nov 28 14:36:55 dev11 rbd-target-api[37651]: Setting up iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 14:36:55 dev11 rbd-target-api[37651]: (Gateway.load_config) successfully loaded existing target definition Nov 28 17:01:40 dev11 systemd[1]: Stopping Ceph iscsi target configuration API... ``` journalctl: ``` Nov 28 17:00:01 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:01 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:04 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:04 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:06 dev11 ceph-mgr[3184]: [172.16.1.3:57002] [GET] [500] [45.074s] [admin] [513.0B] /api/health/minimal Nov 28 17:00:06 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "68eed46b-3ece-4e60-bc17-a172358f2d76"} '] Nov 28 17:00:06 dev11 ceph-mgr[3184]: [172.16.1.3:60128] [GET] [500] [45.070s] [admin] [513.0B] /api/health/minimal Nov 28 17:00:06 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "5b6fdaa2-dc70-48a7-b01f-ca554ecfec41"} '] Nov 28 17:00:07 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:07 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:11 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:11 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:11 dev11 ceph-mgr[3184]: ::ffff:127.0.0.1 - - [28/Nov/2020:17:00:11] "GET /metrics HTTP/1.1" 200 151419 "" "Prometheus/2.7.2" Nov 28 17:00:14 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:14 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:17 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:17 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:20 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:20 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:22 dev11 ceph-mgr[3184]: [172.16.1.3:59834] [GET] [500] [45.062s] [admin] [513.0B] /api/health/minimal Nov 28 17:00:22 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "1ba61331-1dfd-43e7-8ced-9f28aeb8a39c"} '] Nov 28 17:00:23 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:23 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:26 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:26 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:26 dev11 ceph-mgr[3184]: ::ffff:127.0.0.1 - - [28/Nov/2020:17:00:26] "GET /metrics HTTP/1.1" 200 151420 "" "Prometheus/2.7.2" Nov 28 17:00:27 dev11 ceph-mgr[3184]: [172.16.1.3:60132] [GET] [500] [45.081s] [admin] [513.0B] /api/health/minimal Nov 28 17:00:27 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "9c1dd49b-07fb-4c49-a033-f5d8d82d9cbe"} '] Nov 28 17:00:29 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:29 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:32 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:32 dev11 kernel: iSCSI Login negotiation failed. Nov 28 17:00:35 dev11 kernel: Unable to locate Target Portal Group on iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Nov 28 17:00:35 dev11 kernel: iSCSI Login negotiation failed. ``` What should I do for this problem?

2 1

RESTful manager module deprecation
by Ernesto Puerta 30 Nov '20

30 Nov '20

Hi all, Since Ceph-Dashboard RESTful API <https://docs.ceph.com/en/latest/mgr/ceph_api/> is becoming the official RESTful API for Ceph (starting Pacific release), the proposal is to mark RESTful Module <https://docs.ceph.com/en/latest/mgr/restful/> for deprecation by Pacific and for removal on Q-release. You may find a detailed feature-gap analysis in this tracker issue <https://tracker.ceph.com/issues/47066>. We'd like to know about existing users of the RESTful module and their concerns and suggestions for this proposal. Thank you! Kind regards, Ernesto

1 0

Re: rbd image backup best practice
by Janne Johansson 30 Nov '20

30 Nov '20

Den fre 27 nov. 2020 kl 23:21 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > Is there a best practice or guide for backuping rbd images? > One would think that most things that apply to an iscsi mounted device would be equally valid for an RBD mount, so you might look into that for hints and tips on how to backup remote network data devices if you are seeing this from the mounting client point of view. If now, it probably matters what you are aiming for since "backup" is quite a wide concept apart from "copy of my data". Is it best practice for storing sparse images for conserving backup space, quick restores, valid images, legal archiving demands, or "want to try this weird update and be able to move backwards 30 minutes" ? The solution will be quite different depending on what the problem is, more than perhaps "what the mountpoint type is". -- May the most significant bit of your life be positive.

2 1

Re: Manual bucket resharding problem
by Mateusz Skała 30 Nov '20

30 Nov '20

Thank You for response, how I can upload this to metadata? Is this operation safe? Regards Mateusz Skała W dniu sob., 21.11.2020 o 18:01 Amit Ghadge <amitg.b14(a)gmail.com> napisał(a): > I go through this and you need to update bucket metadata, radosgw-admin > metadata get bucket.instance:bucket:xxx > bucket.json, update two parameter > I don't remember but it's look reshard: false and next_marker set empty. > > -AmitG > On Sat, 21 Nov, 2020, 2:04 PM Mateusz Skała, <mateusz.skala(a)gmail.com> > wrote: > >> Hello Community. >> I need Your help. Few days ago I started manual resharding of one bucket >> with large objects. Unfortunately I interrupted this by Ctrl+c. At now I >> can’t start this process again. >> There is message: >> # radosgw-admin bucket reshard --bucket objects --num-shards 2 >> ERROR: the bucket is currently undergoing resharding and cannot be added >> to the reshard list at this time >> >> But list of reshard process is empty: >> # radosgw-admin reshard list >> [] >> >> # radosgw-admin reshard status --bucket objects >> [ >> { >> "reshard_status": "not-resharding", >> "new_bucket_instance_id": "", >> "num_shards": -1 >> } >> ] >> >> How can I fix this situation ? How to restore possibility resharding this >> bucket? >> And BTW is resharding process locking writes/reads on bucket? >> Regards >> Mateusz Skała >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

2 3

rbd image backup best practice
by Marc Roos 30 Nov '20

30 Nov '20

Is there a best practice or guide for backuping rbd images?

2 1

DB sizing for lots of large files
by Richard Thornton 29 Nov '20

29 Nov '20

Hi, Sorry to bother you all. It’s a home server setup. Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB 7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from eMMC). Only CephFS, I'm anticipating having 50-200K files when the 50TB (4+2 EC) is full. I'm trying to address the issue of really big OSD's, not my words, a Redditor: "When you write an object to a drive with collocated db and raw space, the disk has to read/write to both sections before acking a write. That's a lot to ask a 7200 disk to handle gracefully. I believe Red Hat only supports up to 8TB because of performance concerns with larger disks. I may be wrong, but once you are shuffling through 6-10TB of data I'd think those disks are gonna be bogged down in seek time." So I want to have my two DB's on the Optane to avoid the above, am I making sense? OK, so large files have lower metadata overhead than small files. This is for a media library this probably means super low overhead, one guy I spoke to had similar setup and for 48TB used he had a 2.6GB DB? Is there a rough CephFS calculation (each file uses x bytes of metadata), I think I should be safe with 30GB, now I read I should double that (you should allocate twice the size of the biggest layer to allow for compaction) but I only have 118GB and two OSDs so I will have to go for 59GB (or whatever will fit)? I'm thinking that I might not even use 3GB but to be safe I’ll make it 30GB, lets say it settles at 5GB, when compaction comes that means I will only need 20GB and therefore no spillover? I realise that if the Optane dies both OSD's go with it, do I have to configure anything special there (CRUSH would just handle it)? Being that this a home deployment will Ceph be OK with an occasional power outage, I mean if a bluray mkv gets corrupted ripping it again is an easy fix, also I will backup the clusters data once a month? Below are the commands I got from the website: Create the volume groups: $ vgcreate ceph-block-0 /dev/sda $ vgcreate ceph-block-1 /dev/sdb Create the logical volumes: $ lvcreate -l 100%FREE -n block-0 ceph-block-0 $ lvcreate -l 100%FREE -n block-1 ceph-block-1 Create db logical volumes (118GB Optane) $ vgcreate ceph-db-0 /dev/sdc $ lvcreate -L 59GB -n db-0 ceph-db-0 $ lvcreate -L 59GB -n db-1 ceph-db-0 Create the OSDs $ ceph-volume lvm create --bluestore --data ceph-block-0/block-0 --block.db ceph-db-0/db-0 $ ceph-volume lvm create --bluestore --data ceph-block-1/block-1 --block.db ceph-db-0/db-1 Thanks. Richard

5 5

Access/Delete RGW user with leading whitespace
by Benjamin.Zieglmeier 27 Nov '20

27 Nov '20

Hello, In our environment we have a user that has a leading whitespace in the UID. I don’t know how it was created, however I am unable to GET or DELETE it either using `radosgw-admin` or the Admin API: # radosgw-admin user list | grep rgw " rgw-prometheus", "rgw-prometheus", When I try to get info of the user directly, I get: # radosgw-admin user info --uid=" rgw-prometheus" could not fetch user info: no user info saved I’ve tried using the admin API as well while URL encoding the whitespace to %20 and get “Invalid Argument”. This user is not important in any way, however it creates issues trying to monitor the rgw usage logs using https://github.com/blemmenes/radosgw_usage_exporter I’ve considered modifying the script to ignore that user, but clearly Ceph is having troubles addressing it as well, so I figured I’d try to get to the bottom of how to remove this user. We are running 12.2.11 currently, however this cluster was built on 12.2.5 and I’m 99% certain the user was created in 12.2.5 Thanks, Ben

1 1

Tracing in ceph
by Seena Fallah 27 Nov '20

27 Nov '20

Hi all, Does this project work with the latest zipkin apis? https://github.com/ceph/babeltrace-zipkin Also what do you prefer to trace requests for rgw and rbd in ceph? Thanks.

2 3

Public Swift yielding errors since 14.2.12
by Jukka Nousiainen 27 Nov '20

27 Nov '20

Hi all, In reference to: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2KTC7RXQYW… We are seeing similar behavior with public Swift bucket access being broken. In this case RadosGW Nautilus integrated to OpenStack Queens Keystone. Public Swift containers have worked fine from Luminous era up to Nautilus 14.2.11, and started to break when upgrading RadosGW to 14.2.12 or newer. Unsure if this is related to the backport of "rgw: Swift API anonymous access should 401 (pr#37438", or some other rgw change within 14.2.12. I believe the following ceph.conf we use is relevant: rgw_swift_account_in_url = true rgw_keystone_implicit_tenants = false As well as the configured endpoint format: https://fqdn:443/swift/v1/AUTH_%(tenant_id)s Steps to reproduce: Horizon: -------- 1) Public container access - Create a container with "Container Access" set to Public - Click on the Horizon provided Link which is of the format https://fqdn/swift/v1/AUTH_projectUUID/public-test-container/ Expected result: Empty bucket listing Actual result: "AccessDenied" 2) Public object access - Upload an object to the public container - Try to access the object via unauthenticated browser session Expected result: Object downloaded or loaded into browser Actual result: "NoSuchBucket" Also getting similar behavior with Swift CLI tools (ACL '.r:*') from what I can see. Any suggestions how to troubleshoot further? Happy to provide more debug log and configuration details if need be, as well as pointers if something might be actually wrong in our configuration. Also, apologies for the possible double post - I tried to first submit via the hyperkitty web form but that post seems to have gone into a black hole. BR, Jukka

2 2

Re: Manual bucket resharding problem
by Mateusz Skała 27 Nov '20

27 Nov '20

Hi. Thank You, I will try this solution probably on Sunday. Will write results here. Regards Mateusz Skała > Wiadomość napisana przez Amit Ghadge <amitg.b14(a)gmail.com> w dniu 24.11.2020, o godz. 11:12: > > Sorry for delay reply, I never tried on production but after revert the changes I can able to reshard again > You get same bucket two entries from metadata, > radosgw-admin metadata list bucket.instance | grep bucket > You now the older bucket entry then update those two parameter first, > radosgw-admin metadata get bucket.instance:bucket:<id> > bucket.json > Set reshard_status to 0 and new_bucket_instance_id to "" > Update bucket instance by, radosgw-admin metadata put bucket.instance:bucket:<id> < bucket.json > > On Sun, Nov 22, 2020 at 6:04 PM Mateusz Skała <mateusz.skala(a)gmail.com <mailto:mateusz.skala@gmail.com>> wrote: > Thank You for response, how I can upload this to metadata? Is this operation safe? > Regards > Mateusz Skała > > W dniu sob., 21.11.2020 o 18:01 Amit Ghadge <amitg.b14(a)gmail.com <mailto:amitg.b14@gmail.com>> napisał(a): > I go through this and you need to update bucket metadata, radosgw-admin metadata get bucket.instance:bucket:xxx > bucket.json, update two parameter I don't remember but it's look reshard: false and next_marker set empty. > > -AmitG > On Sat, 21 Nov, 2020, 2:04 PM Mateusz Skała, <mateusz.skala(a)gmail.com <mailto:mateusz.skala@gmail.com>> wrote: > Hello Community. > I need Your help. Few days ago I started manual resharding of one bucket with large objects. Unfortunately I interrupted this by Ctrl+c. At now I can’t start this process again. > There is message: > # radosgw-admin bucket reshard --bucket objects --num-shards 2 > ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time > > But list of reshard process is empty: > # radosgw-admin reshard list > [] > > # radosgw-admin reshard status --bucket objects > [ > { > "reshard_status": "not-resharding", > "new_bucket_instance_id": "", > "num_shards": -1 > } > ] > > How can I fix this situation ? How to restore possibility resharding this bucket? > And BTW is resharding process locking writes/reads on bucket? > Regards > Mateusz Skała > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 0

OSD Memory usage
by Seena Fallah 27 Nov '20

27 Nov '20

Hi all, After I upgrade from 14.2.9 to 14.2.14 my OSDs are using less more memory than before! I give each OSD 6GB memory target and before the free memory was 20GB and now after 24h from the upgrade I have 104GB free memory of 128GB memory! Also, my OSD latency got increases! This happens in both SSD and HDD tier. Are there any notes from the upgrade I missed? Is it related to bluefs_buffered_io? If BlueFS do a direct IO shouldn't BlueFS/Bluestore use the targeted memory for its cache and does it mean before the upgrade the memory used was by a kernel that buffers the IO and wasn't for the ceph-osd? Thanks.

2 4

ceph df: pool stored vs bytes_used -- raw or not?
by Dan van der Ster 26 Nov '20

26 Nov '20

Hi, Depending on which cluster I look at (all running v14.2.11), the bytes_used is reporting raw space or stored bytes variably. Here's a 7 year old cluster: # ceph df -f json | jq .pools[0] { "name": "volumes", "id": 4, "stats": { "stored": 1229308190855881, "objects": 294401604, "kb_used": 1200496280133, "bytes_used": 1229308190855881, "percent_used": 0.4401889145374298, "max_avail": 521125025021952 } } Note that stored == bytes_used for that pool. (this is a 3x replica pool). But here's a newer cluster (installed recently with nautilus) # ceph df -f json | jq .pools[0] { "name": "volumes", "id": 1, "stats": { "stored": 680977600893041, "objects": 163155803, "kb_used": 1995736271829, "bytes_used": 2043633942351985, "percent_used": 0.23379847407341003, "max_avail": 2232457428467712 } } In the second cluster, bytes_used is 3x stored. Does anyone know why these are not reported consistently? Noticing this just now, I'll update our monitoring to plot stored rather than bytes_used from now on. Thanks! Dan

2 13

Advice on SSD choices for WAL/DB?
by Andrei Mikhailovsky 26 Nov '20

26 Nov '20

Hello, We are planning to perform a small upgrade to our cluster and slowly start adding 12TB SATA HDD drives. We need to accommodate for additional SSD WAL/DB requirements as well. Currently we are considering the following: HDD Drives - Seagate EXOS 12TB SSD Drives for WAL/DB - Intel D3 S4510 960GB or Intel D3 S4610 960GB Our cluster isn't hosting any IO intensive DBs nor IO hungry VMs such as Exchange, MSSQL, etc. From the documentation that I've read the recommended size for DB is between 1% and 4% of the size of the osd. Would 2% figure be sufficient enough (so around 240GB DB size for each 12TB osd?) Also, from your experience, which is a better model for the SSD DB/WAL? Would Intel S4510 be sufficient enough for our purpose or would the S4610 be a much better choice? Are there any other cost effective performance to consider instead of the above models? The same question to the HDD. Any other drives we should consider instead of the Seagate EXOS series? Thanks for you help and suggestions. Andrei

4 4

octupus: stall i/o during recovery
by Peter Lieven 26 Nov '20

26 Nov '20

Hi, I am currently evaluating ceph and stumbled across an odd issue when an osd comes back online. The osd was taken offline, but is still "in" and is brought back online before it is marked "out". As a test I run a fio job with 4k rand I/O on a 10G rbd volume during the OSD down and up procedure. As OSDs I use 8x 960GB SAS SSDs on 4 Nodes interconnected with 2x10GE each. Network and SSD seems not to be congested at any time. From time to time I see complete stall in the fio benchmark for approx. 10 seconds while recovery is ongoing. All recovery parameteres (max_recovery, max_backfill, sleep etc.) do not seem to influence it. While digging deeper I found that the requests are hanging in the "waiting for readable" state. Help how to debug this further would be great. Might it be linked to the new feature in Octopus to read from all OSDs and not just the primary? Thank you, Peter

1 0

snap permission denied
by vcjouni 26 Nov '20

26 Nov '20

Hi, I have k8s cephfs-provisioner.yaml and storageclass.yaml: --- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: cephfs namespace: cephfs provisioner: ceph.com/cephfs parameters: monitors: 10.32.121.51:6789,10.32.121.52:6789,10.32.121.53:6789 adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: cephfs claimRoot: /pvc-volumes reclaimPolicy: Retain allowVolumeExpansion: true # ceph version ceph version 15.2.5 I'm trying to create snapshots: /source # cd .snap /source/.snap # mkdir snapshot01 mkdir: can't create directory 'snapshot01': Permission denied /source/.snap # # ceph auth ls shows client permissions: client.kubernetes-dynamic-user-a89595fe-2a80-11eb-8b50-1ec01fe0b788 key: AQAjl7Zf9mFbMxAAcGiyth980XbD1rtcBokqAw== caps: [mds] allow r,allow rw path=/pvc-volumes/kubernetes/kubernetes-dynamic-pvc-a89595cc-2a80-11eb-8b50-1ec01fe0b788 caps: [mon] allow r caps: [osd] allow rw pool=cephfs.cephfsvol1.data namespace=fsvolumens_kubernetes-dynamic-pvc-a89595cc-2a80-11eb-8b50-1ec01fe0b788 If I understood correctly that above permission is lacking 's' flag. How can I enable that permission when client is dynamic as above? Is there a way to get around this problem, e.g. how can I define client setup so that it wouldn't be dynamic and I could give required permissions to that? I'm trying to have a backup Pod to generate snapshots and copy those to nfs mount. Br, Jouni

1 0

Re: Unable to reshard bucket
by Eric Ivancich 26 Nov '20

26 Nov '20

Hey Timothy, Did you ever resolve this issue, and if so, how? > Thank you..I looked through both logs and noticed this in the cancel one: > > osd_op(unknown.0.0:4164 41.2 41:55b0279d:reshard::reshard.0000000009:head [call > rgw.reshard_remove] snapc 0=[] ondisk+write+known_if_redirected e24984) v8 -- > 0x7fe9b3625710 con 0 > osd_op_reply(4164 reshard.0000000009 [call] v24984'105796943 uv105796922 ondisk = -2 > ((2) No such file or directory)) v8 ==== 162+0+0 (203651653 0 0) 0x7fe9880044a0 con > 0x7fe9b3625b70 > ERROR: failed to remove entry from reshard log, oid=reshard.0000000009 tenant= bucket=foo > > Is there anything else that I should look for? It looks like the cancel process thinks > that reshard.0000000009 is present (and probably blocking my attempts at resharding) but > it's not actually there and thus can't be removed. Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA

3 4

November Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 25 Nov '20

25 Nov '20

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday November 25th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20201125 <https://pad.ceph.com/p/Ceph_Science_User_Group_20201125> <https://pad.ceph.com/p/Ceph_Science_User_Group_20200923> We try to keep it to an hour or less. Ceph calendar event details: November 25th, 2020 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

2 1

Re: Misleading error (osd has already bound to class) when starting osd on nautilus?
by David Caro 25 Nov '20

25 Nov '20

Forwarding here in case anyone is seeing the same/similar issue, Amit gave really good pointers and a workaround :) Thanks Amit! On 11/25 16:08, Amit Ghadge wrote: > Yes, and if you want avoid in future update this flag to 0 by $echo 0 > > /sys/block/sdx/queue/rotational > > Thanks > > On Wed, Nov 25, 2020 at 4:03 PM David Caro <dcaro(a)wikimedia.org> wrote: > > > > > Yep, you are right: > > > > ``` > > # cat /sys/block/sdd/queue/rotational > > 1 > > ``` > > > > I was looking to the code too but you got there before me :) > > > > https://github.com/ceph/ceph/blob/25ac1528419371686740412616145703810a561f/… > > > > > > It might be an issue with the driver then reporting the wrong data. I'll > > look > > into it. > > > > Do you mind if I reply on the list with this info? (or if you want you > > reply) > > I think this might help others too (and myself in the future xd) > > > > Thanks Amit! > > > > On 11/25 15:50, Amit Ghadge wrote: > > > This might happen when the disk default sets 1 > > > in /sys/block/sdx/queue/rotational , 1 for HDD and 0 for SSD, But we not > > > see any problem till now. > > > > > > -AmitG > > > > > > On Wed, Nov 25, 2020 at 3:08 PM David Caro <dcaro(a)wikimedia.org> wrote: > > > > > > > > > > > Hi! > > > > > > > > I have a nautilus ceph cluster, and today I restarted one of the osd > > > > daemons > > > > and spend some time trying to debug an error I was seeing in the log, > > > > though it > > > > seems the osd is actually working. > > > > > > > > > > > > The error I was seeing is: > > > > ``` > > > > Nov 25 09:07:43 osd15 systemd[1]: Starting Ceph object storage daemon > > > > osd.44... > > > > Nov 25 09:07:43 osd15 systemd[1]: Started Ceph object storage daemon > > > > osd.44. > > > > Nov 25 09:07:47 osd15 ceph-osd[12230]: 2020-11-25 09:07:47.846 > > > > 7f55395fbc80 -1 osd.44 106947 log_to_monitors {default=true} > > > > Nov 25 09:07:47 osd15 ceph-osd[12230]: 2020-11-25 09:07:47.850 > > > > 7f55395fbc80 -1 osd.44 106947 mon_cmd_maybe_osd_create fail: 'osd.44 > > has > > > > already bound to class 'ssd', can not reset class to 'hdd'; use 'ceph > > osd > > > > crush rm-device-class <id>' to remove old class first': (16) Device or > > > > resource busy > > > > ``` > > > > > > > > There's no other messages in the journal so at first I thought that > > the osd > > > > failed to start. > > > > But it seems to be up and working correctly anyhow. > > > > > > > > There's no "hdd" class in my crush map: > > > > ``` > > > > # ceph osd crush class ls > > > > [ > > > > "ssd" > > > > ] > > > > ``` > > > > > > > > And that osd is actually of the correct class: > > > > ``` > > > > # ceph osd crush get-device-class osd.44 > > > > ssd > > > > ``` > > > > > > > > ``` > > > > # uname -a > > > > Linux osd15 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 > > (2020-06-07) > > > > x86_64 GNU/Linux > > > > > > > > # ceph --version > > > > ceph version 14.2.5-1-g23e76c7aa6 > > > > (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable) > > > > ``` > > > > > > > > The osd shows up in the cluster and it's receiving load, so there > > seems to > > > > be > > > > no problem, but does anyone know what that error is about? > > > > > > > > > > > > Thanks! > > > > > > > > > > > > -- > > > > David Caro > > > > SRE - Cloud Services > > > > Wikimedia Foundation <https://wikimediafoundation.org/> > > > > PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 > > > > > > > > "Imagine a world in which every single human being can freely share in > > the > > > > sum of all knowledge. That's our commitment." > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > -- > > David Caro > > SRE - Cloud Services > > Wikimedia Foundation <https://wikimediafoundation.org/> > > PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 > > > > "Imagine a world in which every single human being can freely share in the > > sum of all knowledge. That's our commitment." > > -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."

2 1

DocuBetter Meeting Today
by John Zachary Dover 25 Nov '20

25 Nov '20

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on Nov 26, 2020 at 0200 UTC, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 (Since this particular instance of this meeting will be held on the Wednesday before United States Thanksgiving, I expect a light turnout.) Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 26 Nov 2020 0200 UTC Etherpad: https://pad.ceph.com/p/Ceph_Documentation Meeting: https://bluejeans.com/908675367 Thanks, everyone. Zac Dover

1 0

uniform and list crush bucket algorithm usage in data centers
by Bobby 25 Nov '20

25 Nov '20

Hi all, For placement purposes ceph uses the default straw2 bucket algorithm. I am curious if the other two bucket algorithms like uniform and list are also being used in some present use cases in data centers? Are there any use cases where straw2 is not being used at all ? BR

1 0

KeyError: 'targets' when adding second gateway on ceph-iscsi - BUG
by Hamidreza Hosseini 25 Nov '20

25 Nov '20

Hi, I have installed ceph-iscsi on ubuntu 20 manully, But when I want to add second gateway it will show me error: ``` OS: Ubuntu 20 LTS ceph version : octopus I install cluster with ceph-ansible but I install ceph-iscsi manually via following link: https://docs.ceph.com/en/latest/rbd/iscsi-target-cli/ (instead of 'yum install ceph-iscsi' I commit 'apt install ceph-iscsi') root@dev13:~# gwcli -v gwcli - 2.7 ``` ``` /iscsi-target...-igw/gateways> create ceph-gateway1 192.168.200.33 Adding gateway, sync'ing 0 disk(s) and 0 client(s) KeyError: 'targets' ``` and it will jump out of gwcli! this is my gwcli log: ``` root@dev13:~# cat gwcli.log 2020-11-25 14:24:56,440 DEBUG [ceph.py:32:__init__()] Adding ceph cluster 'ceph' to the UI 2020-11-25 14:24:57,049 DEBUG [ceph.py:241:populate()] Fetching ceph osd information 2020-11-25 14:24:57,086 DEBUG [ceph.py:150:update_state()] Querying ceph for state information 2020-11-25 14:24:57,197 DEBUG [storage.py:105:refresh()] Refreshing disk information from the config object 2020-11-25 14:24:57,197 DEBUG [storage.py:108:refresh()] - Scanning will use 8 scan threads 2020-11-25 14:24:57,254 DEBUG [storage.py:135:refresh()] - rbd image scan complete: 0s 2020-11-25 14:24:57,254 DEBUG [gateway.py:378:refresh()] Refreshing gateway & client information 2020-11-25 14:24:57,254 DEBUG [ceph.py:150:update_state()] Querying ceph for state information 2020-11-25 14:24:57,294 DEBUG [ceph.py:260:refresh()] Gathering pool stats for cluster 'ceph' 2020-11-25 14:25:02,319 DEBUG [ceph.py:32:__init__()] Adding ceph cluster 'ceph' to the UI 2020-11-25 14:25:03,076 DEBUG [ceph.py:241:populate()] Fetching ceph osd information 2020-11-25 14:25:03,168 DEBUG [ceph.py:150:update_state()] Querying ceph for state information 2020-11-25 14:25:03,219 DEBUG [storage.py:105:refresh()] Refreshing disk information from the config object 2020-11-25 14:25:03,219 DEBUG [storage.py:108:refresh()] - Scanning will use 8 scan threads 2020-11-25 14:25:03,273 DEBUG [storage.py:135:refresh()] - rbd image scan complete: 0s 2020-11-25 14:25:03,274 DEBUG [gateway.py:378:refresh()] Refreshing gateway & client information 2020-11-25 14:25:03,274 DEBUG [ceph.py:150:update_state()] Querying ceph for state information 2020-11-25 14:25:03,314 DEBUG [ceph.py:260:refresh()] Gathering pool stats for cluster 'ceph' 2020-11-25 14:25:32,950 DEBUG [ceph.py:32:__init__()] Adding ceph cluster 'ceph' to the UI 2020-11-25 14:25:33,614 DEBUG [ceph.py:241:populate()] Fetching ceph osd information 2020-11-25 14:25:33,652 DEBUG [ceph.py:150:update_state()] Querying ceph for state information 2020-11-25 14:25:33,714 DEBUG [storage.py:105:refresh()] Refreshing disk information from the config object 2020-11-25 14:25:33,714 DEBUG [storage.py:108:refresh()] - Scanning will use 8 scan threads 2020-11-25 14:25:33,811 DEBUG [storage.py:135:refresh()] - rbd image scan complete: 0s 2020-11-25 14:25:33,811 DEBUG [gateway.py:378:refresh()] Refreshing gateway & client information 2020-11-25 14:25:33,811 DEBUG [ceph.py:150:update_state()] Querying ceph for state information 2020-11-25 14:25:33,864 DEBUG [ceph.py:260:refresh()] Gathering pool stats for cluster 'ceph' 2020-11-25 14:26:02,665 DEBUG [gateway.py:174:ui_command_create()] CMD: /iscsi create iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw 2020-11-25 14:26:02,666 DEBUG [gateway.py:185:ui_command_create()] Create an iscsi target definition in the UI 2020-11-25 14:26:03,191 INFO [gateway.py:196:ui_command_create()] ok 2020-11-25 14:26:45,455 DEBUG [gateway.py:793:ui_command_create()] CMD: ../gateways/ create dev13 ['192.168.200.23'] nosync=False skipchecks=false 2020-11-25 14:26:45,467 INFO [gateway.py:836:ui_command_create()] Adding gateway, sync'ing 0 disk(s) and 0 client(s) 2020-11-25 14:26:45,949 DEBUG [gateway.py:854:ui_command_create()] Gateway creation successful 2020-11-25 14:26:45,949 DEBUG [gateway.py:855:ui_command_create()] Adding gw to UI 2020-11-25 14:26:45,968 DEBUG [gateway.py:934:refresh()] - checking iSCSI/API ports on dev13 2020-11-25 14:26:45,979 INFO [gateway.py:874:ui_command_create()] ok 2020-11-25 14:28:41,572 DEBUG [ceph.py:32:__init__()] Adding ceph cluster 'ceph' to the UI 2020-11-25 14:28:42,269 DEBUG [ceph.py:241:populate()] Fetching ceph osd information 2020-11-25 14:28:42,297 DEBUG [ceph.py:150:update_state()] Querying ceph for state information 2020-11-25 14:28:42,334 DEBUG [storage.py:105:refresh()] Refreshing disk information from the config object 2020-11-25 14:28:42,334 DEBUG [storage.py:108:refresh()] - Scanning will use 8 scan threads 2020-11-25 14:28:42,378 DEBUG [storage.py:135:refresh()] - rbd image scan complete: 0s 2020-11-25 14:28:42,378 DEBUG [gateway.py:378:refresh()] Refreshing gateway & client information 2020-11-25 14:28:42,449 DEBUG [gateway.py:934:refresh()] - checking iSCSI/API ports on dev13 2020-11-25 14:28:42,463 DEBUG [ceph.py:150:update_state()] Querying ceph for state information 2020-11-25 14:28:42,517 DEBUG [ceph.py:260:refresh()] Gathering pool stats for cluster 'ceph' 2020-11-25 14:28:52,659 DEBUG [gateway.py:793:ui_command_create()] CMD: ../gateways/ create ceph-gateway1 ['192.168.200.33'] nosync=False skipchecks=false 2020-11-25 14:28:52,672 INFO [gateway.py:836:ui_command_create()] Adding gateway, sync'ing 0 disk(s) and 0 client(s) 2020-11-25 14:28:53,406 DEBUG [gateway.py:854:ui_command_create()] Gateway creation successful 2020-11-25 14:28:53,407 DEBUG [gateway.py:855:ui_command_create()] Adding gw to UI ``` ``` root@dev13:~# sudo gwcli export copy { "created": "2020/11/25 10:54:39", "discovery_auth": { "mutual_password": "", "mutual_password_encryption_enabled": false, "mutual_username": "", "password": "", "password_encryption_enabled": false, "username": "" }, "disks": {}, "epoch": 2, "gateways": { "dev13": { "active_luns": 0, "created": "2020/11/25 10:56:45", "updated": "2020/11/25 10:56:45" } }, "targets": { "iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw": { "acl_enabled": true, "auth": { "mutual_password": "", "mutual_password_encryption_enabled": false, "mutual_username": "", "password": "", "password_encryption_enabled": false, "username": "" }, "clients": {}, "controls": {}, "created": "2020/11/25 10:56:02", "disks": {}, "groups": {}, "ip_list": [ "192.168.200.23" ], "portals": { "dev13": { "gateway_ip_list": [ "192.168.200.23" ], "inactive_portal_ips": [], "portal_ip_addresses": [ "192.168.200.23" ], "tpgs": 1 } }, "updated": "2020/11/25 10:56:45" } }, "updated": "2020/11/25 10:56:45", "version": 11 } ``` I even changed python3 to python2 but I have the same issue but in the proccess of defining iqn. What should I do for this problem?

1 0

Re: Documentation of older Ceph version not accessible anymore on docs.ceph.com
by John Zachary Dover 25 Nov '20

25 Nov '20

I'll carry this message to the Leadership Team this week and, if Thanksgiving proves an impediment to addressing it, next week. Thanks for raising this concern. Zac Dover Upstream Docs Ceph On Wed, Nov 25, 2020 at 1:43 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > 2nd that. Why even remove old documentation before it is migrated to the > new environment. It should be left online until the migration > successfully completed. > > > > -----Original Message----- > Sent: Tuesday, November 24, 2020 4:23 PM > To: Frank Schilder > Cc: ceph-users > Subject: [ceph-users] Re: Documentation of older Ceph version not > accessible anymore on docs.ceph.com > > I want to just echo this sentiment. I thought the lack of older docs > would be a very temporary issue, but they are still not available. It is > especially frustrating when half the google searches also return a page > not found error. The migration has been very badly done. > > Sincerely, > > On Tue, Nov 24, 2020 at 2:52 AM Frank Schilder <frans(a)dtu.dk> wrote: > > > Older versions are available here: > > > > > > https://web.archive.org/web/20191226012841/https://docs.ceph.com/docs/ > > mimic/ > > > > I'm actually also a bit unhappy about older versions missing. Mimic is > > > not end of life and a lot of people still use luminous. Since there > > are such dramatic differences between interfaces, the old docs should > > not just disappear. > > > > Best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: Dan Mick <dmick(a)redhat.com> > > Sent: 24 November 2020 01:53:29 > > To: Martin Palma > > Cc: ceph-users > > Subject: [ceph-users] Re: Documentation of older Ceph version not > > accessible anymore on docs.ceph.com > > > > I don't know the answer to that. > > > > On 11/23/2020 6:59 AM, Martin Palma wrote: > > > Hi Dan, > > > > > > yes I noticed but now only "latest", "octopus" and "nautilus" are > > > offered to be viewed. For older versions I had to go directly to > > > github. > > > > > > Also simply switching the URL from > > > "https://docs.ceph.com/en/nautilus/" to > > > "https://docs.ceph.com/en/luminous/" will not work any more. > > > > > > Is it planned to make the documentation of the older version > > > available again through doc.ceph.com? > > > > > > Best, > > > Martin > > > > > > On Sat, Nov 21, 2020 at 2:11 AM Dan Mick <dmick(a)redhat.com> wrote: > > >> > > >> On 11/14/2020 10:56 AM, Martin Palma wrote: > > >>> Hello, > > >>> > > >>> maybe I missed the announcement but why is the documentation of > > >>> the older ceph version not accessible anymore on docs.ceph.com > > >> > > >> It's changed UI because we're hosting them on readthedocs.com now. > > > >> See the dropdown in the lower right corner. > > >> > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > > > > -- > Steven Pine > > *E * steven.pine(a)webair.com | *P * 516.938.4100 x > *Webair* | 501 Franklin Avenue Suite 200, Garden City NY, 11530 > webair.com > [image: Facebook icon] <https://www.facebook.com/WebairInc/> [image: > Twitter icon] <https://twitter.com/WebairInc> [image: Linkedin icon] > <https://www.linkedin.com/company/webair> > NOTICE: This electronic mail message and all attachments transmitted > with it are intended solely for the use of the addressee and may contain > legally privileged proprietary and confidential information. If the > reader of this message is not the intended recipient, or if you are an > employee or agent responsible for delivering this message to the > intended recipient, you are hereby notified that any dissemination, > distribution, copying, or other use of this message or its attachments > is strictly prohibited. If you have received this message in error, > please notify the sender immediately by replying to this message and > delete it from your computer. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Ceph on ARM ?
by Kevin Thorpe 25 Nov '20

25 Nov '20

Sorry, I can't compare. Not used Ceph in anger anywhere else. In our case we were looking for Kubernetes on-premises storage and several points led us to the Ambedded solution. I wouldn't expect them to have the sort of throughput of a full size Xeon server but for our immediate purposes that is not really an issue. - It was a turnkey solution with support. We knew very little about either Kubernetes or Ceph and this was the least risk for us. - Size and power. We only have single racks in two datacentres, space is a serious consideration. Ceph is very machine hungry and the alternatives like Softiron even ran to 7U at least. We get 24 micro servers in 3U. Power is only 105W per unit along with little heat. - Cost. These appliances are incredibly inexpensive for the amount of storage they provide. Even the smallest offerings from people like Dell/EMC were both a lot larger and an astonomical amount more expensive. The licencing is purely related to the physical appliances. Hard drives and M.2 cache can be upgraded. You can even run on a single appliance albeit at much reduced resilience and lost capacity. Makes evaluation really inexpensive. - Open source standard. Anything we learn from running these appliances is directly translatable to any Ceph install. Anything we learned on Dell/EMC would be yet more lock in to Dell/EMC. We intend to experiment with Rook in the near future but our inexperience of both Kubernetes and Ceph made this option too risky for the initial stages. If we run Rook properly we think we can be able to co-locate things like databases with their OSD and storage on one server so that performance is optimal while having the management control of Ceph. But the bulk of our data storage is exactly that and doesn't require massive performance. On Wed, 25 Nov 2020 at 09:35, Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > How does ARM compare to Xeon in latency and cluster utilization? > > > >

1 0

Misleading error (osd has already bound to class) when starting osd on nautilus?
by David Caro 25 Nov '20

25 Nov '20

Hi! I have a nautilus ceph cluster, and today I restarted one of the osd daemons and spend some time trying to debug an error I was seeing in the log, though it seems the osd is actually working. The error I was seeing is: ``` Nov 25 09:07:43 osd15 systemd[1]: Starting Ceph object storage daemon osd.44... Nov 25 09:07:43 osd15 systemd[1]: Started Ceph object storage daemon osd.44. Nov 25 09:07:47 osd15 ceph-osd[12230]: 2020-11-25 09:07:47.846 7f55395fbc80 -1 osd.44 106947 log_to_monitors {default=true} Nov 25 09:07:47 osd15 ceph-osd[12230]: 2020-11-25 09:07:47.850 7f55395fbc80 -1 osd.44 106947 mon_cmd_maybe_osd_create fail: 'osd.44 has already bound to class 'ssd', can not reset class to 'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy ``` There's no other messages in the journal so at first I thought that the osd failed to start. But it seems to be up and working correctly anyhow. There's no "hdd" class in my crush map: ``` # ceph osd crush class ls [ "ssd" ] ``` And that osd is actually of the correct class: ``` # ceph osd crush get-device-class osd.44 ssd ``` ``` # uname -a Linux osd15 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux # ceph --version ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable) ``` The osd shows up in the cluster and it's receiving load, so there seems to be no problem, but does anyone know what that error is about? Thanks! -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."

1 0

S3 Object Lock - ceph nautilus
by Torsten Ennenbach 25 Nov '20

25 Nov '20

We are running our S3 Ceph Cluster on Nautilus 14.2.9 and want to use features like Object-lock-enabled. A creation of a bucket is possible with: aws s3api create-bucket --bucket locktest --endpoint http://our-s3 <http://our-s3/> --object-lock-enabled-for-bucket & aws s3api put-object-lock-configuration --bucket locktest --endpoint http://our-s3 <http://our-s3/> --object-lock-configuration '{ "ObjectLockEnabled“: "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 50 }}}‘ Are still deletable, and we don’t know why. Because this feature was backported to 14.2.5 as you can see here: https://github.com/ceph/ceph/pull/29905 <https://github.com/ceph/ceph/pull/29905> Any idea what we are doing wrong? -- Beste Grüße aus Köln Ehrenfeld Torsten Ennenbach Cloud Architect

1 0

Re: Certificate for Dashboard / Grafana
by E Taka 25 Nov '20

25 Nov '20

FYI: I've found a solution for the Grafana Certificate. Just run the following commands: 1. ceph config-key set mgr/cephadm/grafana_crt -i <cert> ceph config-key set mgr/cephadm/grafana_key -i <key> 2. ceph orch redeploy grafana 3. ceph config set mgr mgr/dashboard/GRAFANA_API_URL https://ceph01.domain.tld:3000

1 0

osd crash: Caught signal (Aborted) thread_name:tp_osd_tp
by Milan Kupcevic 25 Nov '20

25 Nov '20

Hello, Three OSD daemons crash at the same time while processing the same object located in an rbd ec4+2 pool leaving a placement group in inactive down state. Soon after I start the osd daemons back up they crash again choking on the same object. ----------------------------8<------------------------------------ _dump_onode 0x5605a27ca000 4#7:8565da11:::rbd_data.6.a8a8356fd674f.00000000003dce34:head# nid 1889617 size 0x100000 (1048576) expected_object_size 0 expected_write_size 0 in 8 shards, 32768 spanning blobs ----------------------------8<------------------------------------ Please take a look at the attached log file. Ceph status reports: Reduced data availability: 1 pg inactive, 1 pg down Any hints on how to get this placement group back online would be greatly appreciated. Milan -- Milan Kupcevic Senior Cyberinfrastructure Engineer at Project NESE Harvard University FAS Research Computing

2 4

smartctl UNRECOGNIZED OPTION: json=o
by Tony Liu 24 Nov '20

24 Nov '20

Hi, With Ceph Octopus 15.2.5, here is the output of command "ceph device get-health-metrics SEAGATE_DL2400MM0159_WBM2WP2S". =============================== "20201123-000939": { "dev": "/dev/sde", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 231", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "seagate", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\nsmartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-147.el8.x86_64] (local build)\nCopyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org\n\n=======> UNRECOGNIZED OPTION: json=o\n\nUse smartctl -h to get a usage summary\n\n" }, =============================== Is this error expected? The disk is SAS HDD, why those keywords with "nvme_" prefix? Is the same keywords used for all types of disk? Where can I see the command line options for running smartctl? Thanks! Tony

3 2

Ceph on ARM ?
by Adrian Nicolae 24 Nov '20

24 Nov '20

Hi guys, I was looking at some Huawei ARM-based servers and the datasheets are very interesting. The high CPU core numbers and the SoC architecture should be ideal for a distributed storage like Ceph, at least in theory. I'm planning to build a new Ceph cluster in the future and my best case scenario right now is to buy 7 servers with 2 x Intel Silver 12-core or 2 x Gold 20-core CPUs, 32 SATA drives each. And of course - SSDs/NVMEs for wal/db and the metadata pools , 256GB RAM and so on. I'm curious however if the ARM servers are better or not for this use case (object-storage only). For example, instead of using 2xSilver/Gold server, I can use a Taishan 5280 server with 2x Kungpen 920 ARM CPUs with up to 128 cores in total . So I can have twice as many CPU cores (or even more) per server comparing with x86. Probably the price is lower for the ARM servers as well. Has anyone tested Ceph in such scenario ? Is the Ceph software really optimised for the ARM architecture ? What do you think about this ? Thanks !

11 11

Certificate for Dashboard / Grafana
by E Taka 24 Nov '20

24 Nov '20

Hello, I'm having difficulties with setting up the web certificates for the Dashboard on hostnames ceph*01..n*.domain.tld. I set the keys and crt with ceph-config-key. ceph-config-key get mgr/dashbord/crt shows the correct certificate, the same applies to mgr/dashbord/key, mgr/cephadm/grafana_key and mgr/cephadm/grafana_crt. The hosts were rebooted since then, I used different browsers etc. It is no cache or proxy problem. But: 1. The browser complains Grafana web page via https://ceph01.domain.tld:3000 about the certificate and uses the self-signed certificate from cephadm. 2. The Dashboard forwards always from https://ceph01.domain.tld:8443 to https://ceph01:8443, which means that the browser complains again, since it needs the FQDN. How do you handle this? Thanks, Erich

1 0

Unable to find further optimization, or distribution is already perfect
by Toby Darling 24 Nov '20

24 Nov '20

Hi We're having problems getting our erasure coded ec82pool to upmap balance. "ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)": 554 The pool consists of 20 nodes in 10 racks, each rack containing a pair of nodes 1@45*8TB drives and 1@10*16TB. https://pastebin.com/YLwu8VVi The problem is the 8TB drives are roughly 62-74% full, while the 16TB drives are 84-87% full. https://pastebin.com/j7Dx883i Neither osdmaptool or reweight-by-utilization are able to improve the distribution . There's an osdmap ftp://ftp.mrc-lmb.cam.ac.uk/pub/toby/osdmap.2135441. Any thoughts/pointers much appreciated. Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology

2 2

Documentation of older Ceph version not accessible anymore on docs.ceph.com
by Martin Palma 24 Nov '20

24 Nov '20

Hello, maybe I missed the announcement but why is the documentation of the older ceph version not accessible anymore on docs.ceph.com? Best, Martin

6 7

Cephfs snapshots and previous version
by Oliver Weinmann 24 Nov '20

24 Nov '20

Today I played with a samba gateway and cephfs. I couldn’t get previous versions displayed on a windows client and found very little info on the net how to accomplish this. It seems that I need a vfs module called ceph_snapshots. It’s not included in the latest samba version on Centos 8. by this I also noticed that there is no vfs ceph module. Are these modules not stable and therefore not included in centos8? I can compile them but I would like to know why they are not included. And one more question. Are there any plans to add samba gateway support to cephadm? Best regards, Oliver Von meinem iPhone gesendet

4 3

Re: 14. 2.15: Question to collection_list_legacy osd bug fixed in 14.2.15
by Konstantin Shalygin 24 Nov '20

24 Nov '20

No, you are not affected. Affected only clusters with mixed versions. k Sent from my iPhone > On 24 Nov 2020, at 18:25, Rainer Krienke <krienke(a)uni-koblenz.de> wrote: > > Hello, > > thanks for your answer. If I understand you correctly then only if I > upgrade from 14.2.11 to 14.2.(12|14) this could lead to problems. Does > this mean that running 14.2.13 what I have currently installed is not > affected? > > The release note says: > > * BlueStore: Fixes a bug in collection_list_legacy which makes pgs > inconsistent during scrub when running mixed versions of osds, prior to > 14.2.12 with newer. > > This sounds to me like if I create a new OSD having ceph vers < 14.2.15 > (like 14.2.13 in my case) installed and have OSDs created before 14.2.12 > and then create a new one I could be affected by inconsitent pgs during > scrubbing. > > Have a nice day > Rainer > >> Am 24.11.20 um 11:18 schrieb Konstantin Shalygin: >> This bug may be affect when you upgrade from 14.2.11 to 14.2.(12|14) with low speed (e.g. one by one node). If you already upgrade from 14.2.11 you just jump over this bug. >> >> >> k >> >> Sent from my iPhone >> >>>> On 24 Nov 2020, at 10:43, Rainer Krienke <krienke(a)uni-koblenz.de> wrote: >>> >>> Hello, >>> >>> I am running a productive ceph cluster with Nautilus 14.2.13. All OSDs >>> are bluestore and were created with a ceph version prior to 14.2.12. >>> >>> What I would like to know is how urgent I should consider the >>> collection_list_legacy bug since at the moment I am not going to add a >>> brand new OSD to the system. However any time a disk could fail and so I >>> would have to destroy the OSD with the failed disk and them with a new >>> disk run ceph-volume to create a new bluestore OSD. >>> >>> Would this scenario also lead to inconsistent pgs? >>> >>> Thanks >>> Rainer >>> >>>> Am 24.11.20 um 02:35 schrieb David Galloway: >>>> This is the 15th backport release in the Nautilus series. This release >>>> fixes a ceph-volume regression introduced in v14.2.13 and includes few >>>> other fixes. We recommend users to update to this release. >>>> >>>> For a detailed release notes with links & changelog please refer to the >>>> official blog entry at https://ceph.io/releases/v14-2-15-nautilus-released >>>> >>>> >>>> Notable Changes >>>> --------------- >>>> * ceph-volume: Fixes lvm batch --auto, which breaks backward >>>> compatibility when using non rotational devices only (SSD and/or NVMe). >>>> * BlueStore: Fixes a bug in collection_list_legacy which makes pgs >>>> inconsistent during scrub when running mixed versions of osds, prior to >>>> 14.2.12 with newer. >>>> * MGR: progress module can now be turned on/off, using the commands: >>>> `ceph progress on` and `ceph progress off`. >>>> >>>> >>>> Getting Ceph >>>> ------------ >>>> * Git at git://github.com/ceph/ceph.git >>>> * Tarball at http://download.ceph.com/tarballs/ceph-14.2.15.tar.gz >>>> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ >>>> * Release git sha1: afdd217ae5fb1ed3f60e16bd62357ca58cc650e5 >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>> >>> -- >>> Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 >>> 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 >>> Web: http://userpages.uni-koblenz.de/~krienke >>> PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > > > -- > Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 > 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 > Web: http://userpages.uni-koblenz.de/~krienke > PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html

1 0

Prometheus monitoring
by Michael Thomas 24 Nov '20

24 Nov '20

I am gathering prometheus metrics from my (unhealthy) Octopus (15.2.4) cluster and notice a discrepency (or misunderstanding) with the ceph dashboard. In the dashboard, and with ceph -s, it reports 807 million objects objects: pgs: 169747/807333195 objects degraded (0.021%) 78570293/807333195 objects misplaced (9.732%) 24/101158245 objects unfound (0.000%) But in the prometheus metrics (and in ceph df), it reports almost a factor of 10 fewer objects (dominated by pool 7): # HELP ceph_pool_objects DF pool objects # TYPE ceph_pool_objects gauge ceph_pool_objects{pool_id="4"} 3920.0 ceph_pool_objects{pool_id="5"} 372743.0 ceph_pool_objects{pool_id="7"} 86972464.0 ceph_pool_objects{pool_id="8"} 9287431.0 ceph_pool_objects{pool_id="13"} 8961.0 ceph_pool_objects{pool_id="15"} 0.0 ceph_pool_objects{pool_id="17"} 4.0 ceph_pool_objects{pool_id="18"} 206.0 ceph_pool_objects{pool_id="19"} 8.0 ceph_pool_objects{pool_id="20"} 7.0 ceph_pool_objects{pool_id="21"} 22.0 ceph_pool_objects{pool_id="22"} 203.0 ceph_pool_objects{pool_id="23"} 4415522.0 Why are these two values different? How can I get the total number of objects (807 million) from the prometheus metrics? --Mike

1 0

ssd suggestion
by mj 23 Nov '20

23 Nov '20

Hi, We are going to replace our spinning SATA 4GB filestore disks with new 4GB SSD bluestore disks. Our cluster is reading far more than writing. Comparing options, I found the interesting and cheap Micron 5210 ION 3,84TB SSDs. The way we understand it, there is a performance hit, when it comes to continuous writing speeds. But costwise those SSD's are very interesting. (only 450 euro each) Our cluster is only small: consisting of three servers, in a 3/2 redundancy config. I was planning to replace the 8 OSDs on one server, and then take some time to checkout how well (or not...) they perform. We just wanted to ask here: anyone with suggestions on alternative SSDs we should consider? Or other tips we should take into consideration..? Thanks, MJ

2 1

PGs undersized for no reason?
by Frank Schilder 23 Nov '20

23 Nov '20

Hi all, I'm upgrading ceph mimic 13.2.8 to 13.2.10 and make a strange observation. When restarting OSDs on the new version, the PGs come back as undersized. They are missing 1 OSD and I get a lot of objects degraded/misplaced. I have only the noout flag set. Can anyone help me out why the PGs don't peer until they are all complete? Is there a flag I can set to get complete PGs before starting backfill/recovery? Ceph is currently rebuilding objects even though all data should still be there. Hence, the update takes an unreasonable amount of time now and I remember that with the update from 13.2.2 to 13.2.8 PGs came back complete really fast. There was no such extended period with incomplete PGs and degraded redundancy. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

HA_proxy setup
by Szabo, Istvan (Agoda) 23 Nov '20

23 Nov '20

Hi, I wonder is there anybody have a setup like I want to setup? 1st subnet: 10.118.170.0/24 (FE users) 2nd subnet: 10.192.150.0/24 (BE users) The users are coming from these subnets, and I want that the FE users will come on the 1st interface on the loadbalancer, the BE users will come one the 2nd interface of the HA_Proxy loadbalancer, so somehow need to create 2 backends maybe in the HA_Proxy config? Both users would go to the same rados gateways and to the same ceph cluster. Somehow I want to create static routes on the loadbalancer, but not sure how can I define in the HA_Proxy config, that go to that specific interface ? Thanks ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Sizing radosgw and monitor
by Szabo, Istvan (Agoda) 23 Nov '20

23 Nov '20

Hi, I haven't really find any documentation about how to size radosgw. One redhat doc says we need to decide the ratio like 1:50 or 1:100 osd / rgw. I had an issue earlier where I had a user who source loadbalanced so always went to the same radosgateway and 1 time just maxed out. So the question is, how to monitor rgw, what kind of values or ... ? How to size RGW? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

using fio tool in ceph development cluster (vstart.sh)
by Bobby 22 Nov '20

22 Nov '20

Hi, I am using the Ceph development cluster through vstart.sh script. I would like to measure/benchmark read and write performance (benchmark ceph at a low level). For that I want to use the fio tool. Can we use fio on the development cluster? AFAIK, we can..... I have seen the fio option in the CMakeLists.txt of the Ceph source code. Thanks in advance. BR

2 1

v15.2.6 Octopus released
by David Galloway 22 Nov '20

22 Nov '20

This is the 6th backport release in the Octopus series. This releases fixes a security flaw affecting Messenger V2 for Octopus & Nautilus. We recommend users to update to this release. Notable Changes --------------- * CVE 2020-25660: Fix a regression in Messenger V2 replay attacks Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.6.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: cb8c61a60551b72614257d632a574d420064c17a

3 2

question about rgw index pool
by Adrian Nicolae 21 Nov '20

21 Nov '20

Hi guys, I'll have a future Ceph deployment with the following setup : - 7 powerful nodes running Ceph 15.2.x with mon, rgw and osd daemons colocated - 100+ SATA drives with EC 4+2 - every OSD will have a large NVME partition (300GB) for rocksdb - the storage will be dedicated for rgw traffic using Swift (no cephfs, no rbd) - we will probably have a lot of 4MB ceph objects (more than 400k millions) after the first year. Does it still matter to have the rgw index pool on dedicated SSD/NVME drives or it's good enough to spread it on many SATA drives with 3x replication and a large PG number ? Thanks.

1 0

Manual bucket resharding problem
by Mateusz Skała 21 Nov '20

21 Nov '20

Hello Community. I need Your help. Few days ago I started manual resharding of one bucket with large objects. Unfortunately I interrupted this by Ctrl+c. At now I can’t start this process again. There is message: # radosgw-admin bucket reshard --bucket objects --num-shards 2 ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time But list of reshard process is empty: # radosgw-admin reshard list [] # radosgw-admin reshard status --bucket objects [ { "reshard_status": "not-resharding", "new_bucket_instance_id": "", "num_shards": -1 } ] How can I fix this situation ? How to restore possibility resharding this bucket? And BTW is resharding process locking writes/reads on bucket? Regards Mateusz Skała

1 0

Problems with mon
by Mateusz Skała 21 Nov '20

21 Nov '20

Hello Community, I have problems with ceph-mons in docker. Docker pods are starting but I got a lot of messages "e6 handle_auth_request failed to assign global_id” in log. 2 mons are up but I can’t send any ceph commands. Regards Mateusz

2 4

Multisite design details
by Girish Aher 20 Nov '20

20 Nov '20

Hello All, I am looking to understand some of the internal details on how multisite is architected. On the Ceph user list, I see mentions of metadata logs, bucket index shard logs etc. but there is just no documentation anywhere I could find on how multisite works using these. Could someone please point me in the right direction here? Apart from the code, is there any resource that could help me with understanding the multisite internals? --Girish

1 0

The serious side-effect of rbd cache setting
by norman 20 Nov '20

20 Nov '20

Hi All, We're testing the rbd cache setting for openstack(Ceph 14.2.5 Bluestore 3-replica), and an odd problem found: 1. Setting librbd cache [client] rbd cache = true rbd cache size = 16777216 rbd cache max dirty = 12582912 rbd cache target dirty = 8388608 rbd cache max dirty age = 1 rbd cache writethrough until flush = true 2. Running rbd bench rbd -c /etc/ceph/ceph.conf \ -k /etc/ceph/keyring2 \ -n client.rbd-openstack-002 bench \ --io-size 4K \ --io-threads 1 \ --io-pattern seq \ --io-type read \ --io-total 100G \ openstack-volumes/image-you-can-drop-me 3. Start another test rbd -c /etc/ceph/ceph.conf \ -k /etc/ceph/keyring2 \ -n client.rbd-openstack-002 bench \ --io-size 4K \ --io-threads 1 \ --io-pattern rand \ --io-type write \ --io-total 100G \ openstack-volumes/image-you-can-drop-me2 Running for minutes, I found the read test almost hung for a while: 69 152069 2375.21 9728858.72 70 153627 2104.63 8620569.93 71 155748 1956.04 8011953.10 72 157665 1945.84 7970177.24 73 159661 1947.64 7977549.44 74 161522 1890.45 7743277.01 75 163583 1991.04 8155301.58 76 165791 2008.44 8226566.26 77 168433 2153.43 8820438.66 78 170269 2121.43 8689377.16 79 172511 2197.62 9001467.33 80 174845 2252.22 9225091.00 81 177089 2259.42 9254579.83 82 179675 2248.22 9208708.30 83 182053 2356.61 9652679.11 84 185087 2515.00 10301433.50 99 185345 550.16 2253434.96 101 185346 407.76 1670187.73 102 185348 282.44 1156878.38 103 185350 162.34 664931.53 104 185353 12.86 52681.27 105 185357 1.93 7916.89 106 185361 2.74 11235.38 107 185367 3.27 13379.95 108 185375 5.08 20794.43 109 185384 6.93 28365.91 110 185403 9.19 37650.06 111 185438 17.47 71544.17 128 185467 4.94 20243.53 129 185468 4.45 18210.82 131 185469 3.89 15928.44 132 185493 4.09 16764.16 133 185529 4.16 17037.21 134 185578 18.64 76329.67 135 185631 27.78 113768.65 Why this happened? It's a unacceptable performance for read.

2 3

CephFS error: currently failed to rdlock, waiting. clients crashing and evicted
by Thomas Hukkelberg 20 Nov '20

20 Nov '20

Hi all! Hopefully some of you can shed some light on this. We have big problems with samba crashing when macOS smb clients access certain/random folders/files over vfs_ceph. When browsing cephfs folder in question directly on a cephnode where cephfs is mouted we experience some issues like slow dir listing. We suspect that maybe macOS fetching of xattr metadata creates a lot of traffic, but it should not lockup the cluster like this. In logs we see both rdlock and wrlock, but mostly rdlocks. End clients experience spurious disconnects when issue occurs, roughly up to a handfull times a day. Is this a config issue? Have we hit a bug? It's certainly not a feature :/ Any pointers on how to troubleshoot or rectify this problem is most welcome. ceph version 14.2.11 samba version 4.12.10-SerNet-Ubuntu-10.focal Supermicro X11, Intel Silver 4110, 9 ceph nodes, 2x40gbe network, 150OSD spinners, NVMe db/journal -- 2020-11-17 22:09:07.525706 [WRN] evicting unresponsive client bo-samba-03 (3887652779), after 301.746 seconds 2020-11-17 22:09:07.525580 [INF] Evicting (and blacklisting) client session 3877970532 (10.40.30.133:0/3971626932) 2020-11-17 22:09:07.525536 [WRN] evicting unresponsive client bo-samba-03 (3877970532), after 302.034 seconds 2020-11-17 22:07:23.915412 [INF] Cluster is now healthy 2020-11-17 22:07:23.915381 [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2020-11-17 22:07:23.915330 [INF] Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1 clients failing to respond to capability release) 2020-11-17 22:07:23.064492 [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 2020-11-17 22:07:23.064457 [INF] MDS health message cleared (mds.?): Client bo-samba-03 failing to respond to capability release 2020-11-17 22:07:17.524023 [WRN] client.3887663354 isn't responding to mclientcaps(revoke), ino 0x10001202b55 pending pAsLsXsFs issued pAsLsXsFsx, sent 63.325997 seconds ago 2020-11-17 22:07:17.523987 [INF] Evicting (and blacklisting) client session 3887663354 (10.40.30.133:0/3230547239) 2020-11-17 22:07:17.523967 [WRN] evicting unresponsive client bo-samba-03 (3887663354), after 64.5412 seconds 2020-11-17 22:07:17.523610 [WRN] slow request 63.325528 seconds old, received at 2020-11-17 22:06:14.197986: client_request(client.3878823430:4 lookup #0x100011f9a68/mappe uten navn 2020-11-17 22:06:14.197908 caller_uid=111139, caller_gid=110513{}) currently failed to rdlock, waiting 2020-11-17 22:07:17.523596 [WRN] 1 slow requests, 1 included below; oldest blocked for > 63.325529 secs 2020-11-17 22:07:19.255177 [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) 2020-11-17 22:07:12.523453 [WRN] 1 slow requests, 0 included below; oldest blocked for > 58.325433 secs 2020-11-17 22:07:07.523382 [WRN] 1 slow requests, 0 included below; oldest blocked for > 53.325362 secs 2020-11-17 22:07:02.523360 [WRN] 1 slow requests, 0 included below; oldest blocked for > 48.325307 secs 2020-11-17 22:06:57.523218 [WRN] 1 slow requests, 0 included below; oldest blocked for > 43.325199 secs 2020-11-17 22:06:52.523203 [WRN] 1 slow requests, 0 included below; oldest blocked for > 38.325158 secs 2020-11-17 22:06:47.523105 [WRN] slow request 33.325065 seconds old, received at 2020-11-17 22:06:14.197986: client_request(client.3878823430:4 lookup #0x100011f9a68/mappe uten navn 2020-11-17 22:06:14.197908 caller_uid=111139, caller_gid=110513{}) currently failed to rdlock, waiting 2020-11-17 22:06:47.523100 [WRN] 1 slow requests, 1 included below; oldest blocked for > 33.325065 secs 2020-11-17 22:06:51.431745 [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2020-11-17 22:06:20.045030 [INF] Cluster is now healthy 2020-11-17 22:06:20.045008 [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2020-11-17 22:06:20.044960 [INF] Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1 clients failing to respond to capability release) 2020-11-17 22:06:19.062307 [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 2020-11-17 22:06:19.062253 [INF] MDS health message cleared (mds.?): Client bo-samba-03 failing to respond to capability release 2020-11-17 22:06:15.936150 [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) 2020-11-17 22:06:12.522624 [WRN] client.3869410498 isn't responding to mclientcaps(revoke), ino 0x10001202b55 pending pAsLsXsFs issued pAsLsXsFsx, sent 64.045677 seconds ago --thomas -- Thomas Hukkelberg thomas(a)hovedkvarteret.no +47 971 81 192 -- support(a)hovedkvarteret.no +47 966 44 999

2 1

one osd down / rgw damoen wont start
by Bernhard Krieger 20 Nov '20

20 Nov '20

Hello, today i came across a strange behaviour. After stoppping an osd, im not able to restart or /stop/start a radosgw daemon. The boot proccess will stuck until i have started the osd again. Specs: 3 ceph nodes 2 radosgw nautilus 14.2.13 CentOS7 Steps: * stopping radosgw daemon on rgw * stopping one osd on a ceph-node * starting radosgw daemon on rgw * rgw daemon stucks in boot proccess 2020-11-20 09:58:35.412 7f8a0b7c4900 0 framework: civetweb 2020-11-20 09:58:35.412 7f8a0b7c4900 0 framework conf key: port, val: 10.220.196.31:80 2020-11-20 09:58:35.412 7f8a0b7c4900 0 framework conf key: num_threads, val: 100 2020-11-20 09:58:35.412 7f8a0b7c4900 0 deferred set uid:gid to 167:167 (ceph:ceph) 2020-11-20 09:58:35.412 7f8a0b7c4900 0 ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable), process radosgw, pid 8145 -- > STUCKS/no log entries * As soon as i starting the osd, the boot sequence continues and everything works. 2020-11-20 09:58:35.412 7f8a0b7c4900 0 framework: civetweb 2020-11-20 09:58:35.412 7f8a0b7c4900 0 framework conf key: port, val: 10.220.196.31:80 2020-11-20 09:58:35.412 7f8a0b7c4900 0 framework conf key: num_threads, val: 100 2020-11-20 09:58:35.412 7f8a0b7c4900 0 deferred set uid:gid to 167:167 (ceph:ceph) 2020-11-20 09:58:35.412 7f8a0b7c4900 0 ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable), process radosgw, pid 8145 2020-11-20 10:00:23.895 7f8a0b7c4900 0 starting handler: civetweb 2020-11-20 10:00:23.913 7f8a0b7c4900 1 mgrc service_daemon_register rgw.rgw1 metadata {arch=x86_64,ceph_release=nautilus,ceph_version=ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable),ceph_version_short=14.2.13,cpu=Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS),distro=centos,distro_description=CentOS Linux 7 (Core),distro_version=7,frontend_config#0=civetweb port=10.220.196.31:80 num_threads=100,frontend_type#0=civetweb,hostname=rgw1,kernel_description=#1 SMP Tue Oct 20 16:53:08 UTC 2020,kernel_version=3.10.0-1160.2.2.el7.x86_64,mem_swap_kb=1048572,mem_total_kb=1881836,num_handles=1,os=Linux,pid=8145,zone_id=ecf200a8-2c4a-4c96-96d8-4fcff5b2c8c3,zone_name=default,zonegroup_id=01d11ed1-6157-4c26-addf-ecba49820e20,zonegroup_name=default} 2020-11-20 10:00:25.551 7f89d3aac700 1 civetweb: 0x557f6a6f2000: 10.220.199.4 - - [20/Nov/2020:10:00:25 +0100] "GET / HTTP/1.0" 200 416 - - 2020-11-20 10:00:25.582 7f89d3aac700 1 civetweb: 0x557f6a6f2000: 10.220.199.3 - - [20/Nov/2020:10:00:25 +0100] "GET / HTTP/1.0" 200 416 - - 2020-11-20 10:00:27.555 7f89d3aac700 1 civetweb: 0x557f6a6f2000: 10.220.199.4 - - [20/Nov/2020:10:00:27 +0100] "GET / HTTP/1.0" 200 416 - - 2020-11-20 10:00:27.586 7f89d3aac700 1 civetweb: 0x557f6a6f2000: 10.220.199.3 - - [20/Nov/2020:10:00:27 +0100] "GET / HTTP/1.0" 200 416 - - regards Bernhard

2 1

EC cluster cascade failures and performance problems
by Paul Kramme 19 Nov '20

19 Nov '20

Hello, currently, we are experiencing problems with a cluster used for storing RBD backups. Config: * 8 nodes, each with 6 HDDs OSDs and 1 SSD used for blockdb and WAL * k=4 m=2 EC * dual 25GbE NIC * v14.2.8 ceph health detail shows the following messages: HEALTH_WARN BlueFS spillover detected on 1 OSD(s); 45 pgs not deep-scrubbed in time; snap trim queue for 2 pg(s) >= 32768 (mon_osd_snap_trim_queue_warn_on); 1 slow ops, oldest one blocked for 18629 sec, mon.cloud10-1517 has slow ops BLUEFS_SPILLOVER BlueFS spillover detected on 1 OSD(s) osd.0 spilled over 68 MiB metadata from 'db' device (35 GiB used of 185 GiB) to slow device PG_NOT_DEEP_SCRUBBED 45 pgs not deep-scrubbed in time pg 18.3f5 not deep-scrubbed since 2020-09-03 21:58:28.316958 pg 18.3ed not deep-scrubbed since 2020-09-01 15:11:54.335935 [--- cut ---] PG_SLOW_SNAP_TRIMMING snap trim queue for 2 pg(s) >= 32768 (mon_osd_snap_trim_queue_warn_on) snap trim queue for pg 18.2c5 at 41630 snap trim queue for pg 18.d6 at 44079 longest queue on pg 18.d6 at 44079 try decreasing "osd snap trim sleep" and/or increasing "osd pg max concurrent snap trims". SLOW_OPS 1 slow ops, oldest one blocked for 18629 sec, mon.cloud10-1517 has slow ops We've made some observations on that cluster: * The BlueFS spillover goes away with "ceph tell osd.0 compact" but comes back eventually * The blockdb/WAL SSD is highly utilized, while the HDDs are not * When one OSD fails, there is a cascade failure taking down many other OSDs across all nodes. Most of the time, the cluster comes back when setting the nodown flag and restarting all failed OSDs one by one * Sometimes, especially during maintenance, "Long heartbeat ping times on front/back interface seen, longest is 1390.076 msec" messages pop up * The cluster performance deteriorates sharply when upgrading from 14.2.8 to 14.2.11 or later, so we've rolled back to 14.2.8 Of these problems, the OSD cascade failure is the most important, and is responsible for lenghty downtimes in the past few weeks. Do you have any ideas on how to combat these problems? Thank you, Paul -- Mit freundlichen Grüßen Paul Kramme Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Stefan Priebe, Marc Zocher, Dr. Claus Boyens, Daniel Hagemeier Aufsichtsrat: Gabriele Pulvermüller (Vorsitzende)

2 2

Mon's falling out of quorum, require rebuilding. Rebuilt with only V2 address.
by Wesley Dillingham 19 Nov '20

19 Nov '20

We have had multiple clusters experiencing the following situation over the past few months on both 14.2.6 and 14.2.11. On a few instances it seemed random , in a second situation we had temporary networking disruption, in a third situation we accidentally made some osd changes which caused certain OSDs to reach the hard limit of too many pgs-per-osd and said PGs were stuck inactive. Regardless of the scenario one monitor always falls out of quorum and seemingly can't rejoin and its logs contain the following: 2020-11-18 05:34:54.113 7f14286d9700 1 mon.a2plcephmon01@2(probing) e30 handle_auth_request failed to assign global_id 2020-11-18 05:34:54.295 7f14286d9700 -1 mon.a2plcephmon01@2(probing) e30 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2020-11-18 05:34:55.397 7f14286d9700 1 mon.a2plcephmon01@2(probing) e30 handle_auth_request failed to assign global_id Ultimately, we take the quick route of rebuilding the mon by wiping its DB and re-doing a mkfs and rejoining the quorum using: sudo -u ceph ceph-mon -i a2plcephmon01 --public-addr 10.1.1.1:3300 Luckily we have never dropped below 50% in quorum so far but we are very interested in preventing this from happening going forward. Inspecting "ceph mon dump" on the given clusters I see that all of the rebuilt mons use only msgrv2 on 3300 but all of the mons that never required rebuilding are using v1 and v2 addressing. So my question are: - Does this monitor failure sound familiar? - Is it problematic the manner in which we are rebuilding mons and it only lands them on using msgr v2 or should we be using v2 only on an all-nautilus cluster? Thanks for any insight. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

1 0

newbie Cephfs auth permissions issues
by Jonathan D. Proulx 19 Nov '20

19 Nov '20

Hi All, I've been using ceph block and object storage for years but just wandering into cephfs now (Nautilus all servers on 14.2.9 ). I created small data and metadata pools, a new filesystem and used: ceph fs authorize <fsname> client.<name> / rw creating two new users to mount it, both can one using fuse (14.2.9) and one using kernel client (Ubuntu 20.04 kernel 5.4.0-53). so far so good, but then it gets "weird" I can perform metadata operations like "mkdir" and "touch" but not actually write any data: testy-mctestface% touch /mnt/cephfs/boo testy-mctestface% echo foo > /mnt/cephfs/boo echo: write error: operation not permitted auth caps look good to me, but seem most likely to be worng: root@ceph-mon0:/ # ceph auth get client.client0 exported keyring for client.client0 [client.client0] key = <SEKRET> caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=<fsname>" is "data" hear supposed to be <fsname> or <data_pool_name>? Presumably it's fsname since that what the "fa authorize" put there and it should know... can anyone see what I'm doing wrong here? Thanks, -Jon

2 1

BLUEFS_SPILLOVER BlueFS spillover detected
by Zhenshi Zhou 19 Nov '20

19 Nov '20

Hi, I have a cluster of 14.2.8. I created OSDs with dedicated PCIE for wal/db when deployed the cluster. I set 72G for db and 3G for wal on each OSD. And now my cluster is in a WARN stats until a long health time. # ceph health detail HEALTH_WARN BlueFS spillover detected on 1 OSD(s) BLUEFS_SPILLOVER BlueFS spillover detected on 1 OSD(s) osd.63 spilled over 33 MiB metadata from 'db' device (1.5 GiB used of 72 GiB) to slow device I lookup on google and find https://tracker.ceph.com/issues/38745 I'm not sure if it's the same issue. How can I deal with this? THANKS

3 4

Can't upgrade from 15.2.5 to 15.2.6... (Cannot calculate service_id: daemon_id='cephfs....')
by Gencer Genç 19 Nov '20

19 Nov '20

Hi, I ran those commands as usual: $ ceph orch host ls Result is as expected by hosts names and addresses. $ ceph orch ls Again, Expected result as before. Then I started to upgrade via this command: $ ceph orch upgrade start --ceph-version 15.2.6 It failed with attached logs. Please see logs.txt for detailed output. It says; orchestrator._interface.OrchestratorError: DaemonDescription: Cannot calculate service_id: daemon_id='cephfs.host-1.shsruc' hostname='host' This cluster have 2 nodes. Then I just ran same command again: $ ceph orch ls Result is NOT as expected. I got; >> Error EINVAL: DaemonDescription: Cannot calculate service_id: daemon_id='cephfs.host-1.shsruc' hostname='host' (See attached logs.) If I restart "host-1" then i can execute "ceph orch ls" command. After I start upgrade it errors again. Interestingly I only have cephfs mounted in node "host". Not "host-1". How can I fix this issue? Thanks, Gencer.

1 1

Slow OSDs
by Kristof Coucke 19 Nov '20

19 Nov '20

Hi, We are having slow osd's... A hot topic to search on it... I've tried to dive as deep as I can, but I need to know which debug setting will help me to dive even deeper... Okay: situation: - After expansion lot's of backfill operations are running spread over the osd's. - max_backfills is set to 1 (meaning only 1 read and 1 write operation might take place at the same time for the backfill operation) - Disks are monitored and manual compaction is performed as soon as load goes over 80% - Requests thru RGW goes okay for some objects, and less okay for others. When it goes fine requests are returned within a few ms, other requests vary from a few ms to >10s. - Ceph version 14.2.11 - Erasure coding configured, 6-3 - Backfill is running at an incredible terrible pace. I assume it's all related Okay, what have I done to navigate down: 1. Removed 1 RGW from the load balancer to increase logging and dive into it 2. Using s3cmd to request the object teaches me that the delay is between the request from the RGW to the OSD and the response. 3. Investigating the osd using iostat doesn't reveal an issue. Load is very low. 3. Next I increased logging on the osd (debug_osd 20/20) and launched a request (debug setting has been immediately reversed)... 4. First you see that it is starting with the block for getting all available shards... so far so good 5. Next we will see multiple blocks like: 2020-11-19 11:53:21.938 7fe90f79c700 15 osd.156 45341 *enqueue_op* 0x559f94d43ce0 prio 127 cost 0 latency 0.000032 epoch 45341 *MOSDECSubOpRead*(1.1e2s0 45341/45313 ECSubRead(tid=191, to_read={1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head=0,28672,0}, subchunks={1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head=[0,1]}, attrs_to_read=)) v3 2020-11-19 11:53:21.938 7fe90678a700 10 osd.156 45341 *dequeue_op* 0x559f94d43ce0 prio 127 cost 0 latency 0.000135 *MOSDECSubOpRead*(1.1e2s0 45341/45313 ECSubRead(tid=191, to_read={1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head=0,28672,0}, subchunks={1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head=[0,1]}, attrs_to_read=)) v3 pg pg[1.1e2s0( v 45295'4373744 (34740'4370700,45295'4373744] local-lis/les=45313/45317 n=1842926 ec=5869/714 lis/c 45313/45313 les/c/f 45317/45322/0 45313/45313/45313) [156,64,146,256,98,67,46,31,100]p156(0) r=0 lpr=45313 crt=45295'4373744 lcod 0'0 mlcod 0'0 active+clean] 2020-11-19 11:53:21.938 7fe90678a700 20 osd.156 45341 share_map osd.156 v2: 192.168.34.57:6892/1464289 45341 2020-11-19 11:53:21.938 7fe90678a700 20 osd.156 45341 should_share_map osd.156 v2:192.168.34.57:6892/1464289 45341 2020-11-19 11:53:21.938 7fe90678a700 10 osd.156 pg_epoch: 45341 pg[1.1e2s0( v 45295'4373744 (34740'4370700,45295'4373744] local-lis/les=45313/45317 n=1842926 ec=5869/714 lis/c 45313/45313 les/c/f 45317/45322/0 45313/45313/45313) [156,64,146,256,98,67,46,31,100]p156(0) r=0 lpr=45313 crt=45295'4373744 lcod 0'0 mlcod 0'0 active+clean] _handle_message: MOSDECSubOpRead(1.1e2s0 45341/45313 ECSubRead(tid=191, to_read={1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head=0,28672,0}, subchunks={1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head=[0,1]}, attrs_to_read=)) v3 2020-11-19 11:53:21.938 7fe90678a700 20 osd.156 pg_epoch: 45341 pg[1.1e2s0( v 45295'4373744 (34740'4370700,45295'4373744] local-lis/les=45313/45317 n=1842926 ec=5869/714 lis/c 45313/45313 les/c/f 45317/45322/0 45313/45313/45313) [156,64,146,256,98,67,46,31,100]p156(0) r=0 lpr=45313 crt=45295'4373744 lcod 0'0 mlcod 0'0 active+clean] handle_sub_read read request=28672 r=28672 len=28672 2020-11-19 11:53:21.938 7fe90678a700 10 osd.156 pg_epoch: 45341 pg[1.1e2s0( v 45295'4373744 (34740'4370700,45295'4373744] local-lis/les=45313/45317 n=1842926 ec=5869/714 lis/c 45313/45313 les/c/f 45317/45322/0 45313/45313/45313) [156,64,146,256,98,67,46,31,100]p156(0) r=0 lpr=45313 crt=45295'4373744 lcod 0'0 mlcod 0'0 active+clean] get_hash_info: Getting attr on 1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head 2020-11-19 11:53:21.938 7fe90678a700 10 osd.156 pg_epoch: 45341 pg[1.1e2s0( v 45295'4373744 (34740'4370700,45295'4373744] local-lis/les=45313/45317 n=1842926 ec=5869/714 lis/c 45313/45313 les/c/f 45317/45322/0 45313/45313/45313) [156,64,146,256,98,67,46,31,100]p156(0) r=0 lpr=45313 crt=45295'4373744 lcod 0'0 mlcod 0'0 active+clean] get_hash_info: not in cache 1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head 2020-11-19 11:53:21.938 7fe90678a700 10 osd.156 pg_epoch: 45341 pg[1.1e2s0( v 45295'4373744 (34740'4370700,45295'4373744] local-lis/les=45313/45317 n=1842926 ec=5869/714 lis/c 45313/45313 les/c/f 45317/45322/0 45313/45313/45313) [156,64,146,256,98,67,46,31,100]p156(0) r=0 lpr=45313 crt=45295'4373744 lcod 0'0 mlcod 0'0 active+clean] get_hash_info: found on disk, size 28672 2020-11-19 11:53:21.938 7fe90678a700 20 osd.156 pg_epoch: 45341 pg[1.1e2s0( v 45295'4373744 (34740'4370700,45295'4373744] local-lis/les=45313/45317 n=1842926 ec=5869/714 lis/c 45313/45313 les/c/f 45317/45322/0 45313/45313/45313) [156,64,146,256,98,67,46,31,100]p156(0) r=0 lpr=45313 crt=45295'4373744 lcod 0'0 mlcod 0'0 active+clean] handle_sub_read: Checking hash of 1:47c1b09f:::44a57332-bad8-47b4-85d6-6d34be960d5f.51962.196_CT%2f20201001%2f82078cfd-a111-4b58-8b79-88a860915c56.PNG:head 2020-11-19 11:53:21.938 7fe90678a700 20 osd.156 45341 share_map_peer 0x559f71ee3400 don't know epoch, doing nothing 2020-11-19 11:53:21.938 7fe90678a700 10 osd.156 45341 dequeue_op 0x559f94d43ce0 finish .... This goes on... The latency is caused between such blocks. E.g.: Last line of a block of logging: ... 2020-11-19 11:53:*22.150* 7fe90678a700 10 osd.156 45341 dequeue_op 0x559f96a5f600 finish 2020-11-19 11:53:*22.150* 7fe90678a700 20 osd.156 op_wq(2) _process empty q, waiting ... Next block of logging: 2020-11-19 11:53:*27.089* 7fe925251700 15 osd.156 45341 enqueue_op 0x559f8f9618c0 prio 127 cost 0 latency 0.000019 epoch 45341 *MOSDECSubOpReadReply*(1.1e2s0 45341/45313 *ECSubReadReply*(tid=191, attrs_read=0)) v2 2020-11-19 11:53:27.089 7fe925251700 20 osd.156 op_wq(2) _enqueue OpQueueItem(1.1e2s0 PGOpItem(op=*MOSDECSubOpReadReply*(1.1e2s0 45341/45313 ECSubReadReply(tid=191, attrs_read=0)) v2) prio 127 cost 0 e45341) 2020-11-19 11:53:27.089 7fe90678a700 20 osd.156 op_wq(2) _process 1.1e2s0 to_process <> waiting <> waiting_peering {} 2020-11-19 11:53:27.089 7fe90678a700 20 osd.156 op_wq(2) _process OpQueueItem(1.1e2s0 PGOpItem(op=*MOSDECSubOpReadReply*(1.1e2s0 45341/45313 ECSubReadReply(tid=191, attrs_read=0)) v2) prio 127 cost 0 e45341) queued 2020-11-19 11:53:27.089 7fe90678a700 20 osd.156 op_wq(2) _process 1.1e2s0 to_process <OpQueueItem(1.1e2s0 PGOpItem(op=MOSDECSubOpReadReply(1.1e2s0 45341/45313 ECSubReadReply(tid=191, attrs_read=0)) v2) prio 127 cost 0 e45341)> waiting <> waiting_peering {} 2020-11-19 11:53:27.089 7fe90678a700 20 osd.156 op_wq(2) _process OpQueueItem(1.1e2s0 PGOpItem(op=*MOSDECSubOpReadReply*(1.1e2s0 45341/45313 *ECSubReadReply*(tid=191, attrs_read=0)) v2) prio 127 cost 0 e45341) pg 0x559f8b5e5000 2020-11-19 11:53:27.089 7fe90678a700 10 osd.156 45341 dequeue_op 0x559f8f9618c0 prio 127 cost 0 latency 0.000092 MOSDECSubOpReadReply(1.1e2s0 45341/45313 ECSubReadReply(tid=191, attrs_read=0)) v2 pg pg[1.1e2s0( v 45295'4373744 (34740'4370700,45295'4373744] local-lis/les=45313/45317 n=1842926 ec=5869/714 lis/c 45313/45313 les/c/f 45317/45322/0 45313/45313/45313) [156,64,146,256,98,67,46,31,100]p156(0) r=0 lpr=45313 crt=45295'4373744 lcod 0'0 mlcod 0'0 active+clean] Now my question is: What is triggering the enqueue_op? And what is triggering this operation ('cause it takes 5 seconds)? Which debug setting can help me find that out? Many thanks for the feedback, Kristof

1 0

EC overwrite
by Szabo, Istvan (Agoda) 19 Nov '20

19 Nov '20

Hi, Is it s problem if ec_overwrite enabled in the data pool? https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-codi… Thanks ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Weird ceph use case, is there any unknown bucket limitation?
by Szabo, Istvan (Agoda) 19 Nov '20

19 Nov '20

Hi, I have a use case where the user would like to have 50000 Buckets. Is it normal for ceph just too much for me? The reason they want this level of granularity is because they might need to clean buckets for a specific subset and not affect others. Bucket format is this: PR_PAGETPYE-<PAGE_TYPE_ID>_DEVICE-<DEVICE_ID>_WHITELABEL-<WHITELABEL_ID>_LANGUAGE-<LANGUAGE_ID>_SUBTYPE-<SUBTYPE_ID>/<OBJECT_ID> Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

MGR restart loop
by Frank Schilder 19 Nov '20

19 Nov '20

Dear cephers, I have a problem with MGR daemons, ceph version mimic-13.2.8. I'm trying to do maintenance on our MON/MGR servers and am through with 2 out of 3. I have MON and MGR collocated on a host, 3 hosts in total. So far, procedure was to stop the deamons on the server and do the maintenance. Now I'd stuck at the last server, because MGR fail-over does not work. The remaining MGR instances go into a restart loop. In an attempt to mitigate this, I stopped all but 1 MGR on a node that is done with maintenance. Everything fine. However, as soon as I stop the last MON I need to do maintenance on, the last remaining MGR goes into a restart loop all by itself. As far as I can see, the MGR does actually not restart, it just gets thrown out of the cluster. Here is a ceph status before stopping mon.ceph-01: [root@ceph-01 ~]# ceph status cluster: id: xxx health: HEALTH_WARN 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-03(active) mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 302 osds: 281 up, 281 in data: pools: 11 pools, 3215 pgs objects: 334.1 M objects, 689 TiB usage: 877 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 3208 active+clean 7 active+clean+scrubbing+deep As soon as I stop mon.ceph-01, all hell breaks loose. Note that mgr.ceph-03 is collocated with mon.ceph-03 and we have quorum between mon.ceph-02 and mon.ceph-03. Here ceph status snapshots after shutting down mon.ceph-01: [root@ceph-01 ~]# ceph status cluster: id: xxx health: HEALTH_WARN 1 pools nearfull 1/3 mons down, quorum ceph-02,ceph-03 services: mon: 3 daemons, quorum ceph-02,ceph-03, out of quorum: ceph-01 mgr: ceph-03(active) mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 302 osds: 281 up, 281 in data: pools: 11 pools, 3215 pgs objects: 334.1 M objects, 689 TiB usage: 877 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 3207 active+clean 8 active+clean+scrubbing+deep [root@ceph-01 ~]# ceph status cluster: id: xxx health: HEALTH_WARN no active mgr 1 pools nearfull 1/3 mons down, quorum ceph-02,ceph-03 services: mon: 3 daemons, quorum ceph-02,ceph-03, out of quorum: ceph-01 mgr: no daemons active mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 302 osds: 281 up, 281 in data: pools: 11 pools, 3215 pgs objects: 334.1 M objects, 689 TiB usage: 877 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 3207 active+clean 8 active+clean+scrubbing+deep [root@ceph-01 ~]# ceph status cluster: id: xxx health: HEALTH_WARN 1 pools nearfull 1/3 mons down, quorum ceph-02,ceph-03 services: mon: 3 daemons, quorum ceph-02,ceph-03, out of quorum: ceph-01 mgr: ceph-03(active, starting) mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 302 osds: 281 up, 281 in data: pools: 11 pools, 3215 pgs objects: 334.1 M objects, 689 TiB usage: 877 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 3207 active+clean 8 active+clean+scrubbing+deep It is cycling through these 3 states and I couldn't find a reason why. The node ceph-01 is not special in any way. Any hint would be greatly appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 2

v14.2.14 Nautilus released
by David Galloway 19 Nov '20

19 Nov '20

This is the 14th backport release in the Nautilus series. This releases fixes a security flaw affecting Messenger V2 for Octopus & Nautilus, among other fixes across components. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-14-nautilus-released Notable Changes --------------- * CVE 2020-25660: Fix a regression in Messenger V2 replay attacks Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.14.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7e94c5afc28f3eaf36151ad1e1457de5f16c4fdf

1 0

CentOS 8, Ceph Octopus, ssh private key
by Mika Saari 18 Nov '20

18 Nov '20

Hi, Installing Ceph Octopus using cephadm. I managed to install ceph-common with cephadm and when trying to add new hosts with "ceph orch host add ceph2" I get error "Error EINVAL: Failed to connect to ceph2 (ceph2). Check that the host is reachable and accepts connection using the cephadm SSH key". I verified that I am able to ssh login to the ceph2 server with ceph private_key like it was described in the error message. But since adding new hosts to the ceph wasn't working, I tried generating the new private key and updating the public keys to the remote servers with: # ceph cephadm clear-key # ceph cephadm generate-key # ceph cephadm get-pub-key > ceph.pub # ceph config-key get mgr/cephadm/ssh_identity_key > ceph.priv # ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph1 # ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2 # ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3 And then testing that the private key really is working: # chmod 600 ceph.priv # ssh -i ceph.priv root@ceph2 At this point ssh works with passwordless login. But still the ceph orch host add ceph2 doesn't work (giving exactly the same error) I also tried restarting the manager with "ceph mgr fail" which was informed somewhere -> no effect. Also tried rebooting the machines -> no effect. Any tips I could still try ? Thank you very much!

2 3

Using rbd-nbd tool in Ceph development cluster
by Bobby 18 Nov '20

18 Nov '20

Hi, I have to use this *rbd-nbd *tool from Ceph. This is part of Ceph source code. Here: https://github.com/ceph/ceph/tree/master/src/tools/rbd_nbd My question is: Can we use this *rbd-nbd* tool in the Ceph cluster? By Ceph cluster I mean the development cluster we build through *vstart.sh* script. I am quite sure we could use it. I have this script running. I can *start* and *stop* the cluster. But I am struggling to use this rbd-nbd tool in the development cluster which we build through vstart.sh script. Looking for help. Thanks.

2 2

MONs unresponsive for excessive amount of time
by Frank Schilder 18 Nov '20

18 Nov '20

Hi all, one of our MONs was down for maintenance for ca. 45 minutes. After this time I started it up again and it joined the cluster. Unfortunately, things did not go as expected. The MON sub-cluster became unresponsive for a bit more than 10 minutes. Admin commands would hang, even if issued directly to a specific monitor via "ceph tell mon.xxx". In addition, our MDS lost connection to the MONs and reported a laggy connection. Consequently, all ceph fs access was frozen for a bit more than 10 minutes as well. From the little I could get out with "ceph daemon mon.xxx mon_status" I could see that the restarted MON was in state "synchronizing" (or similar, its from memory) while the other mons were in quorum. Our cluster is mimic-12.2.8. Somehow, this observation does not fit together with the intended HA of the MON cluster, there should not be any stall at all. My questions: Why do the MONs become unresponsive for such a long time? What are the MONs doing during this time frame? Are there any config options I should look at? Are there any log messages I should hunt for? Any hint is appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Not all OSDs in rack marked as down when the rack fails
by Wido den Hollander 18 Nov '20

18 Nov '20

Hi, I'm investigating an issue where 4 to 5 OSDs in a rack aren't marked as down when the network is cut to that rack. Situation: - Nautilus cluster - 3 racks - 120 OSDs, 40 per rack We performed a test where we turned off the network Top-of-Rack for each rack. This worked as expected with two racks, but with the third something weird happened. From the 40 OSDs which were supposed to be marked as down only 36 were marked as down. In the end it took 15 minutes for all 40 OSDs to be marked as down. $ ceph config set mon mon_osd_reporter_subtree_level rack That setting is set to make sure that we only accept reports from other racks. What we saw in the logs for example: 2020-10-29T03:49:44.409-0400 7fbda185e700 10 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 osd.51 has 54 reporters, 239.856038 grace (20.000000 + 219.856 + 7.43801e-23), max_failed_since 2020-10-29T03:47:22.374857-0400 But osd.51 was still not marked as down after 54 reporters have reported that it is actually down. I checked, no ping or other traffic possible to osd.51. Host is unreachable. Another osd was marked as down, but it took a couple of minutes as well: 2020-10-29T03:50:54.455-0400 7fbda185e700 10 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 osd.37 has 48 reporters, 221.378970 grace (20.000000 + 201.379 + 6.34437e-23), max_failed_since 2020-10-29T03:47:12.761584-0400 2020-10-29T03:50:54.455-0400 7fbda185e700 1 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 we have enough reporters to mark osd.37 down In the end osd.51 was marked as down, but only after the MON decided to do so: 2020-10-29T03:53:44.631-0400 7fbda185e700 0 log_channel(cluster) log [INF] : osd.51 marked down after no beacon for 903.943390 seconds 2020-10-29T03:53:44.631-0400 7fbda185e700 -1 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107104 no beacon from osd.51 since 2020-10-29T03:38:40.689062-0400, 903.943390 seconds ago. marking down I haven't seen this happen before in any cluster. It's also strange that this only happens in this rack, the other two racks work fine. ID CLASS WEIGHT TYPE NAME -1 1545.35999 root default -206 515.12000 rack 206 -7 27.94499 host CEPH2-206-U16 ... -207 515.12000 rack 207 -17 27.94499 host CEPH2-207-U16 ... -208 515.12000 rack 208 -31 27.94499 host CEPH2-208-U16 ... That's how the CRUSHMap looks like. Straight forward and 3x replication over 3 racks. This issue only occurs in rack *207*. Has anybody seen this before or knows where to start? Wido

2 3

Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type'
by Marcelo 18 Nov '20

18 Nov '20

Hello all. I'm trying to deploy the dashboard (Nautilus 14.2.8), and after I run ceph dashboard create-self-signed-cert, the cluster started to show this warning: # ceph health detail HEALTH_ERR Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type' MGR_MODULE_ERROR Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type' Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type' If I set ceph config set mgr mgr/dashboard/ssl false, the error goes away. I tried to manually upload the certs, but I'm still hitting the error. Has anyone experienced something similar? Thanks, Marcelo.

2 2

Unable to clarify error using vfs_ceph (Samba gateway for CephFS)
by Matt Larson 18 Nov '20

18 Nov '20

I am getting an error in the log.smbd from the Samba gateway that I don’t understand and looking for help from anyone who has gotten the vfs_ceph working. Background: I am trying to get a Samba gateway with CephFS working with the vfs_ceph module. I observed that the default Samba package on CentOS 7.7 did not come with the ceph.so vfs_ceph module, so I tried to compile a working Samba version with vfs_ceph. Newer Samba versions have a requirement for GnuTLS >= 3.4.7, which is not an available package on CentOS 7.7 without a custom repository. I opted to build an earlier version of Samba. On CentOS 7.7, I built Samba 4.11.16 with [global] security = user map to guest = Bad User username map = /etc/samba/smbusers log level = 4 load printers = no printing = bsd printcap name = /dev/null disable spoolss = yes [cryofs_upload] public = yes read only = yes guest ok = yes vfs objects = ceph path = /upload kernel share modes = no ceph:user_id = samba.upload ceph:config_file = /etc/ceph/ceph.conf I have a file at /etc/ceph/ceph.conf including: fsid = redacted mon_host = redacted auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx I have an /etc/ceph/client.samba.upload.keyring /w key for the user `samba.upload` However, connecting fails: smbclient \\\\localhost\\cryofs_upload -U guest Enter guest's password: tree connect failed: NT_STATUS_UNSUCCESSFUL The log.smbd gives these errors: Initialising custom vfs hooks from [ceph] [2020/11/11 17:24:37.388460, 3] ../../lib/util/modules.c:167(load_module_absolute_path) load_module_absolute_path: Module '/usr/local/samba/lib/vfs/ceph.so' loaded [2020/11/11 17:24:37.402026, 1] ../../source3/smbd/service.c:668(make_connection_snum) make_connection_snum: SMB_VFS_CONNECT for service 'cryofs_upload' at '/upload' failed: No such file or directory There is an /upload directory for which the samba.upload user has read access to in the CephFS. What does this error mean: ‘no such file or directory’ ? Is it that vfs_ceph isn’t finding `/upload` or is some other file depended by vfs_ceph not been found? I have also tried to specify a local path rather than a CephFS path and will get the same error. Is there any good guide that describes not just the Samba smb.conf, but also what should be in /etc/ceph/ceph.conf, and how to provide the key for the ceph:user_id ? I am really struggling to find good first-hand documentation for this. Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

4 5

Ceph EC PG calculation
by Szabo, Istvan (Agoda) 18 Nov '20

18 Nov '20

Hi, I have this error: I have 36 osd and get this: Error ERANGE: pg_num 4096 size 6 would mean 25011 total pgs, which exceeds max 10500 (mon_max_pg_per_osd 250 * num_in_osds 42) If I want to calculate the max pg in my server, how it works if I have EC pool? I have 4:2 data EC pool, and the others are replicated. These are the pools: pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode warn last_change 597 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 598 flags hashpspool stripe_width 0 application rgw pool 6 'sin.rgw.log' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 599 flags hashpspool stripe_width 0 application rgw pool 7 'sin.rgw.control' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 600 flags hashpspool stripe_width 0 application rgw pool 8 'sin.rgw.meta' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 601 lfor 0/393/391 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 10 'sin.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 602 lfor 0/529/527 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 11 'sin.rgw.buckets.data.old' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 603 flags hashpspool stripe_width 0 application rgw pool 12 'sin.rgw.buckets.data' erasure profile data-ec size 6 min_size 5 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 604 flags hashpspool,ec_overwrites stripe_width 16384 application rgw So how I can calculate the pgs? This is my osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 534.38354 root default -5 89.06392 host cephosd-6s01 36 nvme 1.74660 osd.36 up 1.00000 1.00000 0 ssd 14.55289 osd.0 up 1.00000 1.00000 8 ssd 14.55289 osd.8 up 1.00000 1.00000 15 ssd 14.55289 osd.15 up 1.00000 1.00000 18 ssd 14.55289 osd.18 up 1.00000 1.00000 24 ssd 14.55289 osd.24 up 1.00000 1.00000 30 ssd 14.55289 osd.30 up 1.00000 1.00000 -3 89.06392 host cephosd-6s02 37 nvme 1.74660 osd.37 up 1.00000 1.00000 1 ssd 14.55289 osd.1 up 1.00000 1.00000 11 ssd 14.55289 osd.11 up 1.00000 1.00000 17 ssd 14.55289 osd.17 up 1.00000 1.00000 23 ssd 14.55289 osd.23 up 1.00000 1.00000 28 ssd 14.55289 osd.28 up 1.00000 1.00000 35 ssd 14.55289 osd.35 up 1.00000 1.00000 -11 89.06392 host cephosd-6s03 41 nvme 1.74660 osd.41 up 1.00000 1.00000 2 ssd 14.55289 osd.2 up 1.00000 1.00000 6 ssd 14.55289 osd.6 up 1.00000 1.00000 13 ssd 14.55289 osd.13 up 1.00000 1.00000 19 ssd 14.55289 osd.19 up 1.00000 1.00000 26 ssd 14.55289 osd.26 up 1.00000 1.00000 32 ssd 14.55289 osd.32 up 1.00000 1.00000 -13 89.06392 host cephosd-6s04 38 nvme 1.74660 osd.38 up 1.00000 1.00000 5 ssd 14.55289 osd.5 up 1.00000 1.00000 7 ssd 14.55289 osd.7 up 1.00000 1.00000 14 ssd 14.55289 osd.14 up 1.00000 1.00000 20 ssd 14.55289 osd.20 up 1.00000 1.00000 25 ssd 14.55289 osd.25 up 1.00000 1.00000 31 ssd 14.55289 osd.31 up 1.00000 1.00000 -9 89.06392 host cephosd-6s05 40 nvme 1.74660 osd.40 up 1.00000 1.00000 3 ssd 14.55289 osd.3 up 1.00000 1.00000 10 ssd 14.55289 osd.10 up 1.00000 1.00000 12 ssd 14.55289 osd.12 up 1.00000 1.00000 21 ssd 14.55289 osd.21 up 1.00000 1.00000 29 ssd 14.55289 osd.29 up 1.00000 1.00000 33 ssd 14.55289 osd.33 up 1.00000 1.00000 -7 89.06392 host cephosd-6s06 39 nvme 1.74660 osd.39 up 1.00000 1.00000 4 ssd 14.55289 osd.4 up 1.00000 1.00000 9 ssd 14.55289 osd.9 up 1.00000 1.00000 16 ssd 14.55289 osd.16 up 1.00000 1.00000 22 ssd 14.55289 osd.22 up 1.00000 1.00000 27 ssd 14.55289 osd.27 up 1.00000 1.00000 34 ssd 14.55289 osd.34 up 1.00000 1.00000 This is the crush rules: [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "replicated_nvme", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~nvme" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "replicated_ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -2, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 3, "rule_name": "sin.rgw.buckets.data.new", "ruleset": 3, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~ssd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] So everything else rather than the data pool are on SSD and nvme with replica 3. If I calculate the pg in the ec like 36osd*100/6=600 which means the max pg in the EC pool is 512? But how this affect the SSD replica pools then? This is the EC pool definition: crush-device-class=ssd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 5

Accessing Ceph Storage Data via Ceph Block Storage
by Vaughan Beckwith 17 Nov '20

17 Nov '20

Hi All, I'm not sure if this is the correct place to ask this question, I have tried the channels, but have received very little help there. I am currently very new to Ceph and am investigating it as to a possible replacement for a legacy application which use to provide us with replication. At the moment my company has three servers, two primary servers running Ubuntu and a backup server also running Ubuntu, the two primary servers each host a virtual machine, and it is these virtual machines that the office workers use for shared folder access, email and as a domain server, the office workers are not aware of the underlying linux servers. In the past the legacy software would replicate the running VM files on both primary servers to the backup server. The replication is done at the underlying linux host level and not from within the guest VMs. I was hoping that I could get Ceph to do this as well. From what I have read and I speak under correction, the best Ceph client type for this would be the block access, whereby I would then mount the block and start up the VMs. As I would be running the VMs, as per normal routine, would Ceph then have to retrieve the large VM files from the storage nodes across the lan and bring the data back to the client to run in the VM. Is there an option to cache certain parts of the data on certain clients? Also none of the primary servers as they currently stand have the capacity to run both VMs together, so each primary has a dedicated VM which it runs, the backup server currently keeps replicated copies of both VM images from each primary, the replication is provided by the legacy application. I'm also wondering if I need to get a fourth server, so I have 2 clients and 2 storage nodes. Any suggestions or help would be greatly appreciated. Yours sincerely Vaughan Beckwith Bluesphere Technologies BSC I.T. (Honours) vaughan.beckwith(a)bluesphere.co.za<mailto:vaughan.beckwith@bluesphere.co.za> Telephone: 011 675 6354 Fax: (011) 675 6423

2 1

(Ceph Octopus) Repairing a neglected Ceph cluster - Degraded Data Reduncancy, all PGs degraded, undersized, not scrubbed in time
by seffyroff＠gmail.com 17 Nov '20

17 Nov '20

I've inherited a Ceph Octopus cluster that seems like it needs urgent maintenance before data loss begins to happen. I'm the guy with the most Ceph experience on hand and that's not saying much. I'm experiencing most of the ops and repair tasks for the first time here. Ceph health output looks like this: HEALTH_WARN Degraded data redundancy: 3640401/8801868 objects degraded (41.359%), 128 pgs degraded, 128 pgs undersized; 128 pgs not deep-scrubbed in time; 128 pgs not scrubbed in time Ceph -s output: https://termbin.com/i06u The crush rule 'cephfs.media' is here: https://termbin.com/2klmq So, it seems like all PGs are in a 'warning' state for the main pool, which is erasure coded and 11TiB across 4 OSDs, of which around 6.4TiB is used. The Ceph services themselves seem happy, they're stable and have Quorum. I'm able to access the web panel fine also. The block devices are of different sizes and types (2 large, different sized spinners, and 2 identical SSDs) I would welcome any pointers on what my steps to bring this up to full health may be. If it's undersized, can I simply add another block device/OSD? Or perhaps adjusting config somewhere will get it to rebalance successfully? (the rebalance jobs have been stuck at 0% for weeks) Thank you for your time reading this message.

8 19