November 2019 - ceph-users

Fwd: Broken: caps osd = "profile rbd-read-only"
by Markus Kienast 06 Nov '19

06 Nov '19

In Nautilus (Ubuntu Cloud Archive Train Version) the osd caps profile rbd-read-only seems broken. It is impossible to map a RBD if the user has the following caps: [client.yyy] key = AQBYL8NdHDpnERAAhk8XOKgFNwhUpCo3EMaW3g== caps mgr = "profile rbd" caps mon = "profile rbd" caps osd = "profile rbd-read-only" (also tried with 'caps osd = "profile rbd-read-only pool=rbd") This is what I do and what I get back: root@maas:~# sudo rbd map rbd/ltsp-01 --read-only --name client.yyy -m 10.101.0.24 --keyfile /etc/ceph/ceph.client.yyy.key rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (1) Operation not permitted I can see no reason in dmesg: root@maas:~# dmesg | tail [ 4442.395179] rbd0: p1 [ 4442.396140] rbd: rbd0: capacity 21474836480 features 0x5 [ 4900.499466] libceph: mon0 (1)10.101.0.24:3300 socket closed (con state CONNECTING) [ 4900.884945] libceph: mon0 (1)10.101.0.24:3300 socket closed (con state CONNECTING) [ 4901.876882] libceph: mon0 (1)10.101.0.24:3300 socket closed (con state CONNECTING) [ 4903.892861] libceph: mon0 (1)10.101.0.24:3300 socket closed (con state CONNECTING) [ 4903.959079] libceph: mon1 (1)10.101.0.24:6789 session established [ 4903.959624] libceph: client15954 fsid afdda66a-000c-11ea-824b-00163e31d42c [ 4923.847328] libceph: mon1 (1)10.101.0.24:6789 session established [ 4923.847749] libceph: client15969 fsid afdda66a-000c-11ea-824b-00163e31d42c Using /sys/bus/rbd/add_single_major directly does not work as well: root@maas:~# echo "10.101.0.24 name=client.yyy,secret=AQBYL8NdHDpnERAAhk8XOKgFNwhUpCo3EMaW3g== rbd ltsp-01" | sudo tee /sys/bus/rbd/add_single_major 10.101.0.24 name=client.yyy,secret=AQBYL8NdHDpnERAAhk8XOKgFNwhUpCo3EMaW3g== rbd ltsp-01 tee: /sys/bus/rbd/add_single_major: Operation not permitted Using a client with 'caps osd = "profile rbd"' does work without any problems. System Information: Ubuntu 18.04.3 ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) Kernel: 5.3.0-19-generic (but also tested with 5.0.0-32-generic and 4.15.0-66-generic) The cluster was built with MAAS and JUJU from openstack-charmers stable charms: Ubuntu 18.04.3 ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) Kernel: 4.15.0-66-generic I wanted to file this as a bug report, but my account (trickkiste) has not been approved yet. Probably due to this bug according to jdillaman on IRC: https://tracker.ceph.com/issues/42667 Regards, Markus

1 0

Re: mds crash loop
by Karsten Nielsen 06 Nov '19

06 Nov '19

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Wed 06-11-2019 14:16 Subject: Re: [ceph-users] mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > -----Original message----- > > From: Yan, Zheng <ukernel(a)gmail.com> > > Sent: Wed 06-11-2019 08:15 > > Subject: Re: [ceph-users] mds crash loop > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > CC: ceph-users(a)ceph.io; > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > > > > > Hi, > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > > > > It was running fine for a while but yesterday my mds went into a crash > loop. > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which is running > the > > > same crash loop. > > > > I am running ceph based on https://hub.docker.com/r/ceph/daemon version > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > Yan, Zheng > > > > Yes I have set that and had to move to pastebin.com as debian apperently only > supports 150k > > > > > > https://pastebin.com/Gv7c5h54 > > > > Looks like on-disk root inode is corrupted. have you encountered any > unusually things during the upgrade? > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' , > check if the object is modified before or after the 'luminous -> > 13.2.6' upgrade. The fil was modified before the upgrade. > To fix the corrupted object. Run 'cephfs-data-scan init > --force-init'. Then restart mds. After mds become active, run 'ceph > daemon mds.x scrub_path / force repair' > > > > - Karsten > > > > > > > > > Thanks for any hints > > > > - Karsten > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > >

1 0

Slow write speed on 3-node cluster with 6* SATA Harddisks (~ 3.5 MB/s)
by Hermann Himmelbauer 06 Nov '19

06 Nov '19

Hi, I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and recreated my ceph cluster with a new release (14.2.4 bluestore) - basically hoping to gain some I/O speed. The installation went flawlessly, reading is faster than before (~ 80 MB/s), however, the write speed is still really slow (~ 3,5 MB/s). I wonder if I can do anything to speed things up? My Hardware is as the following: 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), interconnected via Infiniband 40 The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, so this seems fine. I use ceph for RBD only, so my measurement is simply doing a very simple "dd" read and write test within a virtual machine (Debian 8) like the following: read: dd if=/dev/vdb | pv | dd of=/dev/null -> 80 MB/s write: dd if=/dev/zero | pv | dd of=/dev/vdb -> 3.5 MB/s When I do the same on the virtual machine on a disk that is on a NFS storage, I get something about 30 MB/s for reading and writing. If I disable the write cache on all OSD disks via "hdparm -W 0 /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s. Thanks to your help from the list I plan to install a second ceph cluster which is SSD based (Samsung PM1725b) which should be much faster, however, I still wonder if there is any way to speed up my harddisk based cluster? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers) _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

7 9

Re: mgr daemons becoming unresponsive
by Thomas Schneider 06 Nov '19

06 Nov '19

Well, even after restarting the MGR service the relevant log is spoiled with this error messages: 2019-11-06 17:46:22.363 7f81ffdcc700 0 auth: could not find secret_id=3865 2019-11-06 17:46:22.363 7f81ffdcc700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3865 As you can see the secret_id changes. However I have no idea what's the related service for this secret_id. And in my opinion these errors are preventing the MGR from doing it's job: bringing the cluster to Healthy state. Am 06.11.2019 um 17:41 schrieb Mac Wynkoop: > I actually just had some unresponsive mgr daemons. If It happens > again, I'll see if it's the same error if it happens again. Restarting > them fixed the issue. > Mac Wynkoop > > > > > On Wed, Nov 6, 2019 at 8:43 AM Thomas Schneider <74cmonty(a)gmail.com > <mailto:74cmonty@gmail.com>> wrote: > > Hi, > > does anybody get this error messages in MGR log? > 2019-11-06 15:41:44.765 7f10db740700 0 auth: could not find > secret_id=3863 > 2019-11-06 15:41:44.765 7f10db740700 0 cephx: verify_authorizer could > not get service secret for service mgr secret_id=3863 > > > THX > > Am 06.11.2019 um 10:43 schrieb Oliver Freyermuth: > > Hi together, > > > > interestingly, now that the third mon is missing for almost a week > > (those planned interventions always take longer than expected...), > > we get mgr failovers (but without crashes). > > > > In the mgr log, I find: > > > > 2019-11-06 07:50:05.409 7fce8a0dc700 0 client.0 ms_handle_reset on > > v2:10.160.16.1:6800/618072 <http://10.160.16.1:6800/618072> > > ... > > ... the normal churning ... > > ... > > 2019-11-06 07:52:44.113 7fce8a0dc700 -1 mgr handle_mgr_map I was > > active but no longer am > > 2019-11-06 07:52:44.113 7fce8a0dc700 1 mgr respawn e: > > '/usr/bin/ceph-mgr' > > > > In the mon log, I see: > > ... > > 2019-11-06 07:44:11.565 7f1f44453700 4 rocksdb: > > [db/db_impl_files.cc:356] [JOB 225] Try to delete WAL files size > > 10830909, prev total WAL file size 10839895, number of live WAL > files 2. > > > > 2019-11-06 07:44:11.565 7f1f3a43f700 4 rocksdb: > > [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction > > starting > > 2019-11-06 07:44:11.565 7f1f44c54700 4 rocksdb: (Original Log Time > > 2019/11/06-07:44:11.565802) [db/db_impl_compaction_flush.cc:2374] > > [default] Manual compaction from level-0 to level-6 from 'mgrstat .. > > 'mgrstat; will stop at (end) > > ... > > 2019-11-06 07:50:36.734 7f1f3a43f700 4 rocksdb: > > [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction > > starting > > 2019-11-06 07:52:27.046 7f1f4144d700 0 log_channel(cluster) log > [INF] > > : Manager daemon mon001 is unresponsive, replacing it with standby > > daemon mon002 > > ... > > > > There's a lot of compaction going on (probably due to the prolonged > > HEALTH_WARN state, so not really unexpected) > > so I wonder whether the actual cause for identifying the mgr as > > "unresponsive" is the heavy compaction on the mons. > > It will be interesting to see what happens when we finally have the > > third mon back and the cluster becomes healthy again... > > > > Did somebody see something similar after running for a week or more > > with Nautilus on old and slow hardware? > > > > Cheers, > > Oliver > > > > Am 02.11.19 um 18:20 schrieb Oliver Freyermuth: > >> Dear Sage, > >> > >> good news - it happened again, with debug logs! > >> There's nothing obvious to my eye, it's uploaded as: > >> 0b2d0c09-46f3-4126-aa27-e2d2e8572741 > >> It seems the failure was roughly in parallel to me wanting to > access > >> the dashboard. It must have happened within the last ~5-10 > minutes of > >> the log. > >> > >> I'll now go back to "stable operation", in case you need anything > >> else, just let me know. > >> > >> Cheers and all the best, > >> Oliver > >> > >> Am 02.11.19 um 17:38 schrieb Oliver Freyermuth: > >>> Dear Sage, > >>> > >>> at least for the simple case: > >>> ceph device get-health-metrics osd.11 > >>> => mgr crashes (but in that case, it crashes fully, i.e. the > process > >>> is gone) > >>> I have now uploaded a verbose log as: > >>> ceph-post-file: e3bd60ad-cbce-4308-8b07-7ebe7998572e > >>> > >>> One potential cause of this (and maybe the other issues) might be > >>> because some of our OSDs are on non-JBOD controllers and hence are > >>> made by forming a Raid 0 per disk, > >>> so a simple smartctl on the device will not work (but > >>> -dmegaraid,<number> would be needed). > >>> > >>> Now I have both mgrs active again, debug logging on, device health > >>> metrics on again, > >>> and am waiting for them to become silent again. Let's hope the > issue > >>> reappears before the disks run full of logs ;-). > >>> > >>> Cheers, > >>> Oliver > >>> > >>> Am 02.11.19 um 02:56 schrieb Sage Weil: > >>>> On Sat, 2 Nov 2019, Oliver Freyermuth wrote: > >>>>> Dear Cephers, > >>>>> > >>>>> interestingly, after: > >>>>> ceph device monitoring off > >>>>> the mgrs seem to be stable now - the active one still went > silent > >>>>> a few minutes later, > >>>>> but the standby took over and was stable, and restarting the > >>>>> broken one, it's now stable since an hour, too, > >>>>> so probably, a restart of the mgr is needed after disabling > device > >>>>> monitoring to get things stable again. > >>>>> > >>>>> So it seems to be caused by a problem with the device health > >>>>> metrics. In case this is a red herring and mgrs become instable > >>>>> again in the next days, > >>>>> I'll let you know. > >>>> > >>>> If this seems to stabilize things, and you can tolerate > inducing the > >>>> failure again, reproducing the problem with mgr logs cranked up > >>>> (debug_mgr > >>>> = 20, debug_ms = 1) would probably give us a good idea of why the > >>>> mgr is > >>>> hanging. Let us know! > >>>> > >>>> Thanks, > >>>> sage > >>>> > >>>> > > >>>>> Cheers, > >>>>> Oliver > >>>>> > >>>>> Am 01.11.19 um 23:09 schrieb Oliver Freyermuth: > >>>>>> Dear Cephers, > >>>>>> > >>>>>> this is a 14.2.4 cluster with device health metrics enabled - > >>>>>> since about a day, all mgr daemons go "silent" on me after > a few > >>>>>> hours, i.e. "ceph -s" shows: > >>>>>> > >>>>>> cluster: > >>>>>> id: 269cf2b2-7e7c-4ceb-bd1b-a33d915ceee9 > >>>>>> health: HEALTH_WARN > >>>>>> no active mgr > >>>>>> 1/3 mons down, quorum mon001,mon002 > >>>>>> services: > >>>>>> mon: 3 daemons, quorum mon001,mon002 (age 57m), out > >>>>>> of quorum: mon003 > >>>>>> mgr: no daemons active (since 56m) > >>>>>> ... > >>>>>> (the third mon has a planned outage and will come back in a few > >>>>>> days) > >>>>>> > >>>>>> Checking the logs of the mgr daemons, I find some "reset" > >>>>>> messages at the time when it goes "silent", first for the > first mgr: > >>>>>> > >>>>>> 2019-11-01 21:34:40.286 7f2df6a6b700 0 > log_channel(cluster) log > >>>>>> [DBG] : pgmap v1798: 1585 pgs: 1585 active+clean; 1.1 TiB data, > >>>>>> 2.3 TiB used, 136 TiB / 138 TiB avail > >>>>>> 2019-11-01 21:34:41.458 7f2e0d59b700 0 client.0 > ms_handle_reset > >>>>>> on v2:10.160.16.1:6800/401248 <http://10.160.16.1:6800/401248> > >>>>>> 2019-11-01 21:34:42.287 7f2df6a6b700 0 > log_channel(cluster) log > >>>>>> [DBG] : pgmap v1799: 1585 pgs: 1585 active+clean; 1.1 TiB data, > >>>>>> 2.3 TiB used, 136 TiB / 138 TiB avail > >>>>>> > >>>>>> and a bit later, on the standby mgr: > >>>>>> > >>>>>> 2019-11-01 22:18:14.892 7f7bcc8ae700 0 > log_channel(cluster) log > >>>>>> [DBG] : pgmap v1798: 1585 pgs: 166 active+clean+snaptrim, 858 > >>>>>> active+clean+snaptrim_wait, 561 active+clean; 1.1 TiB data, 2.3 > >>>>>> TiB used, 136 TiB / 138 TiB avail > >>>>>> 2019-11-01 22:18:16.022 7f7be9e72700 0 client.0 > ms_handle_reset > >>>>>> on v2:10.160.16.2:6800/352196 <http://10.160.16.2:6800/352196> > >>>>>> 2019-11-01 22:18:16.893 7f7bcc8ae700 0 > log_channel(cluster) log > >>>>>> [DBG] : pgmap v1799: 1585 pgs: 166 active+clean+snaptrim, 858 > >>>>>> active+clean+snaptrim_wait, 561 active+clean; 1.1 TiB data, 2.3 > >>>>>> TiB used, 136 TiB / 138 TiB avail > >>>>>> > >>>>>> Interestingly, the dashboard still works, but presents outdated > >>>>>> information, and for example zero I/O going on. > >>>>>> I believe this started to happen mainly after the third mon > went > >>>>>> into the known downtime, but I am not fully sure if this > was the > >>>>>> trigger, since the cluster is still growing. > >>>>>> It may also have been the addition of 24 more OSDs. > >>>>>> > >>>>>> > >>>>>> I also find other messages in the mgr logs which seem > >>>>>> problematic, but I am not sure they are related: > >>>>>> ------------------------------ > >>>>>> 2019-11-01 21:17:09.849 7f2df4266700 0 mgr[devicehealth] Error > >>>>>> reading OMAP: [errno 22] Failed to operate read op for oid > >>>>>> Traceback (most recent call last): > >>>>>> File "/usr/share/ceph/mgr/devicehealth/module.py", line 396, > >>>>>> in put_device_metrics > >>>>>> ioctx.operate_read_op(op, devid) > >>>>>> File "rados.pyx", line 516, in > >>>>>> rados.requires.wrapper.validate_func > >>>>>> > (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUIL > >>>>>> D/ceph-14.2.4/build/src/pybind/rados/pyrex/rados.c:4721) > >>>>>> File "rados.pyx", line 3474, in rados.Ioctx.operate_read_op > >>>>>> > (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUILD/ceph-14.2.4/build/src/pybind/rados/pyrex/rados.c:36554) > >>>>>> InvalidArgumentError: [errno 22] Failed to operate read op > for oid > >>>>>> ------------------------------ > >>>>>> or: > >>>>>> ------------------------------ > >>>>>> 2019-11-01 21:33:53.977 7f7bd38bc700 0 mgr[devicehealth] > Fail to > >>>>>> parse JSON result from daemon osd.51 () > >>>>>> 2019-11-01 21:33:53.978 7f7bd38bc700 0 mgr[devicehealth] > Fail to > >>>>>> parse JSON result from daemon osd.52 () > >>>>>> 2019-11-01 21:33:53.979 7f7bd38bc700 0 mgr[devicehealth] > Fail to > >>>>>> parse JSON result from daemon osd.53 () > >>>>>> ------------------------------ > >>>>>> > >>>>>> The reason why I am cautious about the health metrics is that I > >>>>>> observed a crash when trying to query them: > >>>>>> ------------------------------ > >>>>>> 2019-11-01 20:21:23.661 7fa46314a700 0 log_channel(audit) log > >>>>>> [DBG] : from='client.174136 -' entity='client.admin' > >>>>>> cmd=[{"prefix": "device get-health-metrics", "devid": "osd.11", > >>>>>> "target": ["mgr", ""]}]: dispatch > >>>>>> 2019-11-01 20:21:23.661 7fa46394b700 0 mgr[devicehealth] > >>>>>> handle_command > >>>>>> 2019-11-01 20:21:23.663 7fa46394b700 -1 *** Caught signal > >>>>>> (Segmentation fault) ** > >>>>>> in thread 7fa46394b700 thread_name:mgr-fin > >>>>>> > >>>>>> ceph version 14.2.4 > (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) > >>>>>> nautilus (stable) > >>>>>> 1: (()+0xf5f0) [0x7fa488cee5f0] > >>>>>> 2: (PyEval_EvalFrameEx()+0x1a9) [0x7fa48aeb50f9] > >>>>>> 3: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] > >>>>>> 4: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] > >>>>>> 5: (PyEval_EvalFrameEx()+0x67bd) [0x7fa48aebb70d] > >>>>>> 6: (PyEval_EvalCodeEx()+0x7ed) [0x7fa48aebe08d] > >>>>>> 7: (()+0x709c8) [0x7fa48ae479c8] > >>>>>> 8: (PyObject_Call()+0x43) [0x7fa48ae22ab3] > >>>>>> 9: (()+0x5aaa5) [0x7fa48ae31aa5] > >>>>>> 10: (PyObject_Call()+0x43) [0x7fa48ae22ab3] > >>>>>> 11: (()+0x4bb95) [0x7fa48ae22b95] > >>>>>> 12: (PyObject_CallMethod()+0xbb) [0x7fa48ae22ecb] > >>>>>> 13: (ActivePyModule::handle_command(std::map<std::string, > >>>>>> boost::variant<std::string, bool, long, double, > >>>>>> std::vector<std::string, std::allocator<std::string> >, > >>>>>> std::vector<long, std::allocator<long> >, std::vector<double, > >>>>>> std::allocator<double> > >, std::less<void>, > >>>>>> std::allocator<std::pair<std::string const, > >>>>>> boost::variant<std::string, bool, long, double, > >>>>>> std::vector<std::string, std::allocator<std::string> >, > >>>>>> std::vector<long, std::allocator<long> >, std::vector<double, > >>>>>> std::allocator<double> > > > > > const&, > >>>>>> ceph::buffer::v14_2_0::list const&, > std::basic_stringstream<char, > >>>>>> std::char_traits<char>, std::allocator<char> >*, > >>>>>> std::basic_stringstream<char, std::char_traits<char>, > >>>>>> std::allocator<char> >*)+0x20e) [0x55c3c1fefc5e] > >>>>>> 14: (()+0x16c23d) [0x55c3c204023d] > >>>>>> 15: (FunctionContext::finish(int)+0x2c) [0x55c3c2001eac] > >>>>>> 16: (Context::complete(int)+0x9) [0x55c3c1ffe659] > >>>>>> 17: (Finisher::finisher_thread_entry()+0x156) > [0x7fa48b439cc6] > >>>>>> 18: (()+0x7e65) [0x7fa488ce6e65] > >>>>>> 19: (clone()+0x6d) [0x7fa48799488d] > >>>>>> NOTE: a copy of the executable, or `objdump -rdS > <executable>` > >>>>>> is needed to interpret this. > >>>>>> ------------------------------ > >>>>>> > >>>>>> I have issued: > >>>>>> ceph device monitoring off > >>>>>> for now and will keep waiting to see if mgrs go silent > again. If > >>>>>> there are any better ideas or this issue is known, let me know. > >>>>>> > >>>>>> Cheers, > >>>>>> Oliver > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> > >>>>>> > >>>>> > >>> > >>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> > >>> > >> > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> > >> > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> >

1 0

Re: mds crash loop
by Karsten Nielsen 06 Nov '19

06 Nov '19

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Wed 06-11-2019 08:15 Subject: Re: [ceph-users] mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > Hi, > > > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > > It was running fine for a while but yesterday my mds went into a crash loop. > > > > I have 1 active and 1 standby mds for my cephfs both of which is running the > same crash loop. > > I am running ceph based on https://hub.docker.com/r/ceph/daemon version > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > Log details are: https://paste.debian.net/1113943/ > > > > please try again with debug_mds=20. Thanks > > Yan, Zheng Yes I have set that and had to move to pastebin.com as debian apperently only supports 150k https://pastebin.com/Gv7c5h54 - Karsten > > > Thanks for any hints > > - Karsten > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

mds crash loop
by Karsten Nielsen 06 Nov '19

06 Nov '19

Hi, Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 It was running fine for a while but yesterday my mds went into a crash loop. I have 1 active and 1 standby mds for my cephfs both of which is running the same crash loop. I am running ceph based on https://hub.docker.com/r/ceph/daemon version v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. Log details are: https://paste.debian.net/1113943/ Thanks for any hints - Karsten

2 1

Balancer configuration fails with Error EINVAL: unrecognized config option 'mgr/balancer/max_misplaced'
by Thomas Schneider 06 Nov '19

06 Nov '19

Hi, I want to adjust balancer throttling and executed this command that returns an error: root@ld3955:~# ceph config set mgr mgr/balancer/max_misplaced .01 Error EINVAL: unrecognized config option 'mgr/balancer/max_misplaced' root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Ceph is healthy, though: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN mon ld5505 is low on available space services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 19h) mgr: ld5507(active, since 23h), standbys: ld5506, ld5505, ld5508 mds: cephfs:1 {0=ld4464=up:active} 1 up:standby osd: 442 osds: 442 up, 442 in data: pools: 6 pools, 8312 pgs objects: 62.07M objects, 237 TiB usage: 710 TiB used, 821 TiB / 1.5 PiB avail pgs: 8312 active+clean io: client: 1.3 KiB/s rd, 913 KiB/s wr, 1 op/s rd, 34 op/s wr Can you please advise howto adjust balancer throttling? THX

2 1

stretch repository only has ceph-deploy
by Rodrigo Severo - Fábrica 05 Nov '19

05 Nov '19

Hi, I'm trying ceph for the first time. I'm trying to use the repository below: deb https://download.ceph.com/debian-nautilus/ stretch main But it seems that this repository only has the ceph-deploy package, not the rest of ceph. Why is that? How can I get all updated nautilus packages? Regards, Rodrigo Severo

2 1

multiple pgs down with all disks online
by Kári Bertilsson 05 Nov '19

05 Nov '19

pgs: 14.377% pgs not active 3749681/537818808 objects misplaced (0.697%) 810 active+clean 156 down 124 active+remapped+backfilling 1 active+remapped+backfill_toofull 1 down+inconsistent when looking at the down pg's all disks are online 41.3db 53775 0 0 0 401643186092 0 0 3044 down 6m 161222'303144 162913:4630171 [32,96,128,115,86,129,113,124,57,109]p32 [32,96,128,115,86,129,113,124,57,109]p32 2019-11-03 Any way to see why the pg is down ?

2 2

Re: mds crash loop
by Karsten Nielsen 05 Nov '19

05 Nov '19

from ceph -w [root@k8s-node-01 /]# ceph -w cluster: id: 571d4bfe-2c5d-45ca-8da1-91dcaf69942c health: HEALTH_WARN 1 filesystem is degraded services: mon: 3 daemons, quorum k8s-node-00,k8s-node-01,k8s-node-02 mgr: k8s-node-01(active) mds: cephfs-1/1/1 up recovery-fs-1/1/1 up {[cephfs:0]=k8s-node-00=up:active,[recovery-fs:0]=k8s-node-02=up:rejoin}, 1 up:standby osd: 6 osds: 6 up, 6 in rgw: 2 daemons active data: pools: 13 pools, 408 pgs objects: 745.2 k objects, 1.8 TiB usage: 3.6 TiB used, 8.7 TiB / 12 TiB avail pgs: 408 active+clean io: client: 3.3 KiB/s rd, 40 KiB/s wr, 1 op/s rd, 0 op/s wr 2019-11-05 09:43:20.168699 mon.k8s-node-00 [INF] daemon mds.k8s-node-02 restarted 2019-11-05 09:43:21.248009 mon.k8s-node-00 [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-11-05 09:43:21.348459 mon.k8s-node-00 [INF] Standby daemon mds.k8s-node-02 assigned to filesystem recovery-fs as rank 0 2019-11-05 09:43:21.349477 mon.k8s-node-00 [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2019-11-05 09:43:32.129516 mon.k8s-node-00 [INF] daemon mds.k8s-node-02 restarted 2019-11-05 09:43:32.171052 mon.k8s-node-00 [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-11-05 09:43:32.237675 mon.k8s-node-00 [INF] Standby daemon mds.k8s-node-02 assigned to filesystem recovery-fs as rank 0 2019-11-05 09:43:32.238331 mon.k8s-node-00 [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2019-11-05 09:43:46.825739 mon.k8s-node-00 [INF] daemon mds.k8s-node-02 restarted 2019-11-05 09:43:47.780044 mon.k8s-node-00 [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-11-05 09:43:47.821611 mon.k8s-node-00 [INF] Standby daemon mds.k8s-node-02 assigned to filesystem recovery-fs as rank 0 2019-11-05 09:43:47.821976 mon.k8s-node-00 [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) ... ... -----Original message----- From: Karsten Nielsen <karsten(a)foo-bar.dk> Sent: Tue 05-11-2019 10:29 Subject: [ceph-users] mds crash loop To: ceph-users(a)ceph.io; > Hi, > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > It was running fine for a while but yesterday my mds went into a crash loop. > > I have 1 active and 1 standby mds for my cephfs both of which is running the > same crash loop. > I am running ceph based on https://hub.docker.com/r/ceph/daemon version > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > Log details are: https://paste.debian.net/1113943/ > > Thanks for any hints > - Karsten > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

[ceph-user] Upload objects failed on FIPS enable ceph cluster
by Amit Ghadge 04 Nov '19

04 Nov '19

Hi All, I'm using ceph-14.2.4 and testing in FIPS enable cluster. Downloading objects are works but ceph raised segmentation exception while uploading. Please help me here. And please provide debugging stage, So I could take in development environment. Thanks, Amit G

1 0

Ceph + Rook Day San Diego - November 18
by Mike Perez 04 Nov '19

04 Nov '19

Hi Cephers, I'm happy to announce the availability of the schedule for Ceph + Rook Day San Diego. Registration for this event will be free until Tuesday 11:59 UTC, so register now: https://ceph.io/cephdays/ceph-rook-day-san-diego-2019/ We still have some open spots in the schedule, but the lineup is already looking great. We are looking for more Rook related topics. If you're interested, you can submit them through our CFP form for the selection committee to review: https://zfrmz.com/hkg8EF9NYb6IvWnvoRUi We currently have SoftIron, Red Hat and SUSE set as sponsors for this event, we'd like to invite more to contact us at events(a)ceph.io: https://ceph.io/wp-content/uploads/2019/11/Ceph-Day-Partner-Sponsorship.pdf -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Run optimizer to create a new plan on specific pool fails
by Thomas Schneider 04 Nov '19

04 Nov '19

Hi, I want to create an optimizer plan on each pool. My cluster has multiple crush roots, and multiple pools each representing a specific drive (HDD, SSD, NVME). Some pools are balanced, some are not. Therefore I want to run optimizer to create a new plan on specific pool. However this fails for any pool with this error message: root@ld3955:~# ceph balancer optimize hdd-plan hdd Error EALREADY: Unable to find further optimization, or pool(s)' pg_num is decreasing, or distribution is already perfect root@ld3955:~# ceph balancer optimize ssd-plan ssd Error EALREADY: Unable to find further optimization, or pool(s)' pg_num is decreasing, or distribution is already perfect root@ld3955:~# ceph balancer optimize hdb_backup-plan hdb_backup Error EALREADY: Unable to find further optimization, or pool(s)' pg_num is decreasing, or distribution is already perfect root@ld3955:~# ceph osd pool ls hdb_backup hdd ssd nvme cephfs_data cephfs_metadata What is causing this error? THX

1 0

RocksDB device selection (performance requirements)
by Huseyin Cotuk 04 Nov '19

04 Nov '19

Hi all, The only recommendation I can find about db device selection is about the capacity (4% of the data disk) on the documents. Is there any suggestions about technical specs like throughput, IOPS and db device per data disk? While designing a specific infrastructure with filestore, we were looking its specs to meet requirements of all the disks behind the journal device. But in bluestore, data is directly written into data device via bluefs adapter while metadata is written into db (RocksDB) device. I know that it depends on the workload, but is there any best practice or recommendation about selection of db device? IMO, using NVME disks that we used for filestore journals as db devices is not meaningful. Because NVME disks have minimal latency and extraordinary throughput and IOPS performance. So I am not sure that DB device needs that kind of performance. So I want to use those NVME disks for a full flash pool, and choose another disks for db device. Any suggestion or recommendation would be appreciated. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

1 0

Balancer is active, but not balancing
by Thomas 03 Nov '19

03 Nov '19

Hi, I activated balancer in order to balance data distribution: root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } However, the data stored on 1.6TB HDD in specific pool "hdb_backup" is not balanced; the range starts with osd.265 size: 1.6 usage: 52.83 reweight: 1.00000 and ends with osd.145 size: 1.6 usage: 80.19 reweight: 1.00000 The affected drives are located on 4 nodes. The result is that not all available disk space is available for usage. I have attached pastebin <https://pastebin.com/dNyEwNR0> with - ceph osd df sorted by usage - ceph osd df tree Please advise how to start balancer to correct data distribution. THX

1 0

iSCSI write performance
by Ryan 02 Nov '19

02 Nov '19

I'm in the process of testing the iscsi target feature of ceph. The cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts with 12 SSD OSDs per host. Some basic testing moving VMs to a ceph backed datastore is only showing 60MB/s transfers. However moving these back off the datastore is fast at 200-300MB/s. What should I be looking at to track down the write performance issue? In comparison with the Nimble Storage arrays I can see 200-300MB/s in both directions. Thanks, Ryan

8 20

Weird blocked OP issue.
by Robert LeBlanc 02 Nov '19

02 Nov '19

We had an OSD host with 13 OSDs fail today and we have a weird blocked OP message that I can't understand. There are no OSDs with blocked ops, just `mon` (multiple times), and some of the rgw instances. cluster: id: 570bcdbb-9fdf-406f-9079-b0181025f8d0 health: HEALTH_WARN 1 large omap objects Degraded data redundancy: 2083023/195702437 objects degraded (1.064%), 880 pgs degraded, 880 pgs undersized 1609 pgs not deep-scrubbed in time 4 slow ops, oldest one blocked for 506699 sec, daemons [mon,sun-gcs02-rgw01,mon,sun-gcs02-rgw02,mon,sun-gcs02-rgw03] have slow ops. services: mon: 3 daemons, quorum sun-gcs02-rgw01,sun-gcs02-rgw02,sun-gcs02-rgw03 (age 6m) mgr: sun-gcs02-rgw02(active, since 5d), standbys: sun-gcs02-rgw03, sun-gcs02-rgw04 osd: 767 osds: 754 up (since 10m), 754 in (since 104m); 880 remapped pgs rgw: 16 daemons active (sun-gcs02-rgw01.rgw0, sun-gcs02-rgw01.rgw1, sun-gcs02-rgw01.rgw2, sun-gcs02-rgw01.rgw3, sun-gcs02-rgw02.rgw0, sun-gcs02-rgw02.rgw1, sun-gcs02-rgw02.rgw2, sun-gcs02-rgw02.rgw3, sun-gcs02-rgw03.rgw0, sun-gcs02-rgw03.rgw1, sun-gcs02-rgw03.rgw2, s un-gcs02-rgw03.rgw3, sun-gcs02-rgw04.rgw0, sun-gcs02-rgw04.rgw1, sun-gcs02-rgw04.rgw2, sun-gcs02-rgw04.rgw3) data: pools: 7 pools, 8240 pgs objects: 19.57M objects, 52 TiB usage: 88 TiB used, 6.1 PiB / 6.2 PiB avail pgs: 2083023/195702437 objects degraded (1.064%) 43492/195702437 objects misplaced (0.022%) 7360 active+clean 868 active+undersized+degraded+remapped+backfill_wait 12 active+undersized+degraded+remapped+backfilling io: client: 150 MiB/s rd, 642 op/s rd, 0 op/s wr recovery: 626 MiB/s, 223 objects/s $ ceph versions { "mon": { "ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)": 754 }, "mds": {}, "rgw": { "ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)": 16 }, "overall": { "ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)": 754, "ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)": 22 } } I restarted one of the monitors and it dropped out of the list only showing 2 blocked ops, but then showed up again a little while later. Any ideas on where to look? Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

1 1

Ceph Health error right after starting balancer
by Thomas Schneider 01 Nov '19

01 Nov '19

Hi, after enabling ceph balancer (with command ceph balancer on) the health status changed to error. This is the current output of ceph health detail: root@ld3955:~# ceph health detail HEALTH_ERR 1438 slow requests are blocked > 32 sec; 861 stuck requests are blocked > 4096 sec; mon ld5505 is low on available space REQUEST_SLOW 1438 slow requests are blocked > 32 sec 683 ops are blocked > 2097.15 sec 436 ops are blocked > 1048.58 sec 191 ops are blocked > 524.288 sec 78 ops are blocked > 262.144 sec 35 ops are blocked > 131.072 sec 11 ops are blocked > 65.536 sec 4 ops are blocked > 32.768 sec osd.62 has blocked requests > 65.536 sec osds 39,72 have blocked requests > 262.144 sec osds 6,19,67,173,174,187,188,269,434 have blocked requests > 524.288 sec osds 8,16,35,36,37,61,63,64,68,73,75,178,186,271,369,420,429,431,433,436 have blocked requests > 1048.58 sec osds 3,5,7,24,34,38,40,41,59,66,69,74,180,270,370,421,432,435 have blocked requests > 2097.15 sec REQUEST_STUCK 861 stuck requests are blocked > 4096 sec 25 ops are blocked > 8388.61 sec 836 ops are blocked > 4194.3 sec osds 2,28,29,32,60,65,181,185,268,368,423,424,426 have stuck requests > 4194.3 sec osds 0,30,70,71,184 have stuck requests > 8388.61 sec I understand that when balancer starts shifting PGs to other OSDs that this caused IO load on the cluster. However I don't understand why this is affecting OSD so heavily. And I don't understand why OSD of specific type (SSD, NVME) suffer although there's no balancing occuring on them. Regards Thomas

3 4

subtrees have overcommitted (target_size_bytes / target_size_ratio)
by Lars Täuber 01 Nov '19

01 Nov '19

Hello everybody! What does this mean? health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away. $ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on $ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data'] Thanks Lars

3 10

librados aysnc I/O takes considerably longer to complete
by Ponnuvel Palaniyappan 01 Nov '19

01 Nov '19

Hi, Is anyone using librados AIO APIs? I seem to have a problem with that where the rados_aio_wait_for_complete() call just waits for a long period of time before it finishes without error. More info on my setup: I am using Ceph 14.2.4 and write 8MB objects. I run my AIO program on 24 nodes at the same time each writing a different data (splits into 8MB objects and ), each data is about 2G. Normally, it takes about 10 mins for all of them to complete. But often one or more nodes takes considerably longer to finish. When looking at the one of those, I mostly see that the IO requests have been submitted and waits at: #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00002aaaaad0c8fa in rados_aio_wait_for_complete () from /cgv/geovation/2/test/ceph/lib/librados.so.2 Then it eventually completes with no errors from rados_aio_wait_for_complete() call. The (pseudo) code looks like: while (data remains to be written) { size_t aio_ops_count = 0; rados_completion_t aio_comp[12]; for (size_t j = 0; j < 12; ++j) { int err = rados_aio_create_completion(NULL, NULL, NULL, &aio_comp[j]); if (err < 0) { cerr << "rados_aio_create_completion: " << strerror(-err) << endl; return 1; } string obj_ = getobjectid(); err = rados_aio_write_full(io, obj_.c_str(), aio_comp[j], read_buf[j], bytes); if (err < 0) { cerr << "rados_write_full: " << strerror(-err) << endl; return 1; } ++aio_ops_count; } for (size_t j = 0; j < aio_ops_count; ++j) { rados_aio_wait_for_complete(aio_comp[j]); int err = rados_aio_get_return_value(aio_comp[j]); // Considerably longer delay here ?? if (err < 0) { cerr << "rados_aio_get_return_value: " << strerror(-err) << endl; return 1; } rados_aio_release(aio_comp[j]); } } I ran under Valgrind and see no issues and also read the data back and checksum it to verify no corruption issues. So everything appears to "work" as expected except for longer delays at times. Wondering if anyone is using the AIO APIs to write objects and had experienced any similar problems. Please let me know if you need further information. (Originally posted this to dev(a)ceph.io and on Daniel's suggestion, I am posting here). Regards, Ponnuvel P -- Regards, Ponnuvel P

1 0

V/v Multiple pool for data in Ceph object
by tuan dung 01 Nov '19

01 Nov '19

hi ceph-users, i have a cluster run ceph object using version 14.2.1. I want to creat 2 pool for bucket data for purposes for security: + one bucket-data pool for public client access from internet (name *zone1.rgw.buckets.data-pub) * + one bucket-data pool for private client access from local network (name *zone1.rgw.buckets.data-pub)* each pool bucket-data has one individual access key: access key public (access pool public) and access key private (access pool private). Can you give me a recomment for this or bestpractice that you've done? what needs to be done? Or give me your best solution for securiy a cluster ceph object with public client access and private client access? Thank you very much Br, ---------------------------------------------- Dương Tuấn Dũng Email: dungdt.aicgroup(a)gmail.com ĐT: 0986153686

2 2