- ceph-users - lists.ceph.io

Multiple outages when disabling scrubbing
by Bryan Stillwell 04 Jun '20

04 Jun '20

The last two days we've experienced a couple short outages shortly after setting both 'noscrub' and 'nodeep-scrub' on one of our largest Ceph clusters (~2,200 OSDs). This cluster is running Nautilus (14.2.6) and setting/unsetting these flags has been done many times in the past without a problem. One thing I've noticed is that on both days right after setting 'noscrub' or 'nodeep-scrub' that a do_prune message shows up in the monitor logs followed by a timeout. About 30 seconds later we start seeing OSDs getting marked down: 2020-06-03 08:06:53.914 7fcc3ed57700 0 mon.p3cephmon004@0(leader) e11 handle_command mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 2020-06-03 08:06:53.914 7fcc3ed57700 0 log_channel(audit) log [INF] : from='client.5773023471 10.2.128.8:0/523139029' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2020-06-03 08:06:54.231 7fcc4155c700 1 mon.p3lcephmon004(a)0(leader).osd e1535232 do_prune osdmap full prune enabled 2020-06-03 08:06:54.318 7fcc3f558700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7fcc3f558700' had timed out after 0 2020-06-03 08:06:54.319 7fcc4055a700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7fcc4055a700' had timed out after 0 2020-06-03 08:06:54.319 7fcc40d5b700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7fcc40d5b700' had timed out after 0 2020-06-03 08:06:54.319 7fcc3fd59700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7fcc3fd59700' had timed out after 0 ... 2020-06-03 08:07:16.049 7fcc3ed57700 1 mon.p3cephmon004(a)0(leader).osd e1535234 prepare_failure osd.736 [v2:10.6.170.130:6816/1294580,v1:10.6.170.130:6817/1294580] from osd.1165 is reporting failure:1 2020-06-03 08:07:16.049 7fcc3ed57700 0 log_channel(cluster) log [DBG] : osd.736 reported failed by osd.1165 2020-06-03 08:07:16.304 7fcc3ed57700 1 mon.p3cephmon004(a)0(leader).osd e1535234 prepare_failure osd.736 [v2:10.6.170.130:6816/1294580,v1:10.6.170.130:6817/1294580] from osd.127 is reporting failure:1 2020-06-03 08:07:16.304 7fcc3ed57700 0 log_channel(cluster) log [DBG] : osd.736 reported failed by osd.127 2020-06-03 08:07:16.693 7fcc3ed57700 1 mon.p3cephmon004(a)0(leader).osd e1535234 prepare_failure osd.736 [v2:10.6.170.130:6816/1294580,v1:10.6.170.130:6817/1294580] from osd.1455 is reporting failure:1 2020-06-03 08:07:16.693 7fcc3ed57700 0 log_channel(cluster) log [DBG] : osd.736 reported failed by osd.1455 2020-06-03 08:07:16.695 7fcc3ed57700 1 mon.p3cephmon004(a)0(leader).osd e1535234 we have enough reporters to mark osd.736 down 2020-06-03 08:07:16.696 7fcc3ed57700 0 log_channel(cluster) log [INF] : osd.736 failed (root=default,rack=S06-06,chassis=S06-06-17,host=p3cephosd386) (3 reporters from different host after 20.389591 >= grace 20.025280) 2020-06-03 08:07:16.696 7fcc3ed57700 1 mon.p3cephmon004(a)0(leader).osd e1535234 prepare_failure osd.1463 [v2:10.7.208.30:6824/3947672,v1:10.7.208.30:6825/3947672] from osd.1455 is reporting failure:1 2020-06-03 08:07:16.696 7fcc3ed57700 0 log_channel(cluster) log [DBG] : osd.1463 reported failed by osd.1455 2020-06-03 08:07:16.758 7fcc3ed57700 1 mon.p3cephmon004(a)0(leader).osd e1535234 prepare_failure osd.1463 [v2:10.7.208.30:6824/3947672,v1:10.7.208.30:6825/3947672] from osd.2108 is reporting failure:1 2020-06-03 08:07:16.758 7fcc3ed57700 0 log_channel(cluster) log [DBG] : osd.1463 reported failed by osd.2108 2020-06-03 08:07:16.800 7fcc3ed57700 1 mon.p3cephmon004(a)0(leader).osd e1535234 prepare_failure osd.1463 [v2:10.7.208.30:6824/3947672,v1:10.7.208.30:6825/3947672] from osd.1166 is reporting failure:1 2020-06-03 08:07:16.800 7fcc3ed57700 0 log_channel(cluster) log [DBG] : osd.1463 reported failed by osd.1166 2020-06-03 08:07:16.835 7fcc4155c700 1 mon.p3cephmon004(a)0(leader).osd e1535234 do_prune osdmap full prune enabled ... Does any one know why setting the no scrubbing flags would cause such an issue? Or if this is a known issue with a fix in 14.2.9 or 14.2.10 (when it comes out)? Thanks, Bryan

1 0

pg-upmap-items
by Thomas Bennett 03 Jun '20

03 Jun '20

Hi, I've been using pg-upmap items both in the ceph balancer and by hand running osdmaptool for a while now (on Ceph 12.2.13). But I've noticed a side effect of up-map-items which can sometimes lead to some unnecessary data movement. My understanding is that the ceph osdmap keeps track of upmap-items that I undo (in my case using the CERN scrip upmap-remapped.py). These can be seen in the osdmap (or osd dump) json output. It looks, for example, like this: "pg_upmap_items": [ { "pgid": "9.10", "mappings": [ { "from": 1761, "to": 6 } ] }, When upmapping pg 9.10 I first need to clear this pg_upmap_item by executing an rm-upmap-item command: ceph rm-pg-upmap-items 9.10 All this does is unmap the from/to osds (here from osd.6 to osd.1761) which is sometimes not useful. I would prefer to rather "forget" this upmap. I.e remove it permanently from pg_upmap_items. Is there any way to do this? Cheers, Toms

1 0

Deploy nfs from cephadm
by Simon Sutter 03 Jun '20

03 Jun '20

Hello Ceph users, I'm trying to deploy nfs-ganesha with cephadm on octopus. According to the docs, it's as easy as running the command in the docs: https://docs.ceph.com/docs/master/cephadm/install/#deploying-nfs-ganesha

4 5

Upgrade: It is NOT safe to stop mon.vx-rg23-rk65-u43-130
by Gencer W. Genç 03 Jun '20

03 Jun '20

Hi All, I am trying to upgrade ceph 15.2.1 to 15.2.3. I've two node setup on small environment for test only. I ran the following commands: $ ceph mon ok-to-stop mon.vx-rg23-rk65-u43-130 >> quorum should be preserved (vx-rg23-rk65-u43-130,vx-rg23-rk65-u43-130-1) after stopping [mon.vx-rg23-rk65-u43-130] $ ceph orch upgrade start --ceph-version 15.2.3 However, Ceph says it is NOT safe to stop mon.vx-rg23-rk65-u43-130. Debug log: 2020-06-01T22:09:21.967310+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [INF] Upgrade: Checking mon daemons... 2020-06-01T22:09:21.967426+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] daemon mon.vx-rg23-rk65-u43-130 not correct (docker.io/ceph/ceph:v15, bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2, 15.2.1) 2020-06-01T22:09:21.967563+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] Have connection to vx-rg23-rk65-u43-130 2020-06-01T22:09:21.967668+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] None container image docker.io/ceph/ceph:v15.2.3 2020-06-01T22:09:21.967778+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] args: --image docker.io/ceph/ceph:v15.2.3 inspect-image 2020-06-01T22:09:23.400842+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] code: 0 2020-06-01T22:09:23.401062+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] out: { "ceph_version": "ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable)", "image_id": "d72755c420bcbdae08d063de6035d060ea0487f8a43f777c75bdbfcd9fd907fa" } 2020-06-01T22:09:23.404700+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] mon_command: 'mon ok-to-stop' -> -16 in 0.002s 2020-06-01T22:09:23.405002+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [INF] Upgrade: It is NOT safe to stop mon.vx-rg23-rk65-u43-130 2020-06-01T22:09:38.416475+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] mon_command: 'mon ok-to-stop' -> -16 in 0.003s 2020-06-01T22:09:38.417296+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [INF] Upgrade: It is NOT safe to stop mon.vx-rg23-rk65-u43-130 2020-06-01T22:09:53.421473+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] mon_command: 'mon ok-to-stop' -> -16 in 0.003s 2020-06-01T22:09:53.422350+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [INF] Upgrade: It is NOT safe to stop mon.vx-rg23-rk65-u43-130 2020-06-01T22:10:08.440422+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [DBG] mon_command: 'mon ok-to-stop' -> -16 in 0.003s 2020-06-01T22:10:08.441122+0000 mgr.vx-rg23-rk65-u43-130-1.pxmyie [INF] Upgrade: It is NOT safe to stop mon.vx-rg23-rk65-u43-130 How can I solve this and upgrade? Thanks, Gencer.

1 1

Re: professional services and support for newest Ceph
by Patrick Calhoun 03 Jun '20

03 Jun '20

Kevin, Ignazio, Marc, Thanks for the information. I now consider myself well-advised. -Patrick On Tue, Jun 2, 2020 at 1:21 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Ceph is from redhat and redhat is owned by IBM. I think the best > training you could get would be from RedHat. > > I would not advise to learn how to use a mouse with a web interface nor > this ansible or some other deploy tool. Do it from scratch manually so > you know the basics. If you know those, go for some tools that make your > life easier. (and never install the newest stable release ;)) > > > > > -----Original Message----- > Subject: [ceph-users] Re: professional services and support for newest > Ceph > > Hello, I am testing ceph from croit and it works fine: very easy web > interface for installing and managing ceph and very clear support > pricing. > Ignazio > > Il Mar 2 Giu 2020, 19:36 <response(a)ifastnet.com> ha scritto: > > > and theres > > > > https://croit.io/consulting > > > > best regards > > Kevin M > > > > ----- Original Message ----- > > From: "Patrick Calhoun" <phineas(a)ou.edu> > > To: ceph-users(a)ceph.io > > Sent: Tuesday, June 2, 2020 5:29:11 PM > > Subject: [ceph-users] professional services and support for newest > > Ceph > > > > Are there reputable training/support options for Ceph that are not > > geared toward a specific commercial product (e.g. "Red Hat Ceph > > Storage,") but instead would cover the newest open source stable > release? > > > > Thanks, > > Patrick > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > > -- Patrick Calhoun, RHCE Petascale Storage Administrator OU Supercomputing Center for Education and Research Department of Information Technology University of Oklahoma (405) 325-4210

1 0

professional services and support for newest Ceph
by Patrick Calhoun 03 Jun '20

03 Jun '20

Are there reputable training/support options for Ceph that are not geared toward a specific commercial product (e.g. "Red Hat Ceph Storage,") but instead would cover the newest open source stable release? Thanks, Patrick

5 5

OSD upgrades
by Brent Kennedy 02 Jun '20

02 Jun '20

We are rebuilding servers and before luminous our process was: 1. Reweight the OSD to 0 2. Wait for rebalance to complete 3. Out the osd 4. Crush remove osd 5. Auth del osd 6. Ceph osd rm # Seems the luminous documentation says that you should: 1. Out the osd 2. Wait for the cluster rebalance to finish 3. Stop the osd 4. Osd purge # Is reweighting to 0 no longer suggested? Side note: I tried our existing process and even after reweight, the entire cluster restarted the balance again after step 4 ( crush remove osd ) of the old process. I should also note, by reweighting to 0, when I tried to run "ceph osd out #", it said it was already marked out. I assume the docs are correct, but just want to make sure since reweighting had been previously recommended. Regards, -Brent Existing Clusters: Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi gateways ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.2 with 11 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.2 with 12 osd servers, 3 mons, 4 gateways US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons, 3 gateways, 2 iscsi gateways

4 4

upgrade ceph and use cephadm - rgw issue
by Andy Goldschmidt 02 Jun '20

02 Jun '20

Hi I am trying to upgrade from Mimic (13.2.10) to Octopus (15.x). Im also tryin to sue cephadm and am following this guide.https://docs.ceph.com/docs/master/cephadm/adoption/ It was all going fine until step 11 and deploying the new RGW's. I don't have any realms set for my cluster, so how do I do it? This is also a single site cluster. # radosgw-admin realm list{ "default_info": "", "realms": []} # radosgw-admin zone list{ "default_info": "a15e2aec-a0da-4cad-a1bd-f448f25bbe3d", "zones": [ "default" ]} The step below is what I don't know what to do, as it says I need to specify the realm. 11. Redeploy RGW daemons. Cephadm manages RGW daemons by zone. For each zone, deploy new RGW daemons with cephadm: # ceph orch apply rgw <realm> <zone> <placement> [--port <port>] [--ssl] my ceph.conf has this in it about the rgw's[client.rgw.ceph-mgmt0]host = ceph-mgmt0keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-mgmt0/keyringlog file = /var/log/ceph/ceph-rgw-ceph-mgmt0.logrgw frontends = civetweb port=10.92.135.40:8080 num_threads=100rgw dns name = library.xxxxx.com [client.rgw.ceph-mgmt1]host = ceph-mgmt1keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-mgmt1/keyringlog file = /var/log/ceph/ceph-rgw-ceph-mgmt1.logrgw frontends = civetweb port=10.92.135.41:8080 num_threads=100rgw dns name = library.xxxxx.com [client.rgw.ceph-mgmt2]host = ceph-mgmt2keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-mgmt2/keyringlog file = /var/log/ceph/ceph-rgw-ceph-mgmt2.logrgw frontends = civetweb port=10.92.135.42:8080 num_threads=100rgw dns name = library.xxxxx.com RegardsAndy

1 0

Radosgw PubSub Traffic
by Dustin Guerrero 02 Jun '20

02 Jun '20

Hey all, We’ve been running some benchmarks against Ceph which we deployed using the Rook operator in Kubernetes. Everything seemed to scale linearly until a point where I see a single OSD receiving much higher CPU load than the other OSDs (nearly 100% saturation). After some investigation we noticed a ton of pubsub traffic in the strace coming from the RGW pods like so: [pid 22561] sendmsg(77, {msg_name(0)=NULL, msg_iov(3)=[{"\21\2)\0\0\0\10\0:\1\0\0\10\0\0\0\0\0\10\0\0\0\0\0\0\20\0\0-\321\211K"..., 73}, {"\200\0\0\0pubsub.user.ceph-user-wwITOk"..., 314}, {"\0\303\34[\360\314\233\2138\377\377\377\377\377\377\377\377", 17}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL|MSG_MORE <unfinished …> I’ve checked other OSDs and only a single OSD receives these messages. I suspect its creating a bottleneck. Does anyone have an idea on why these are being generated or how to stop them? The pubsub sync module doesn’t appear to be enabled, and our benchmark is doing simple gets/puts/deletes. We’re running Ceph 14.2.5 nautilus Thank you!

2 2

Re: Thread::try_create(): pthread_create failed
by 展荣臻（信泰） 02 Jun '20

02 Jun '20

Thank your reply Our cluster are runing for two years in production,and it has no problem,so we don't upgrade. I check memory on host.Very little memory of free left.Does creating thread failure have anything to do with this? In addition to the kvm virtual machine, there are 22 osds on the host. free -m total used free shared buff/cache available Mem: 515420 178212 4323 729 332884 335360 Swap: 8191 8145 46 > sysctl: > kernel.pid_max=4194303 > kernel.threads-max=2097152 > vm.max_map_count=524288 > > But really, why are you still running Hammer? Later releases handle a large number of OSDs *much* better. > > > On Jun 1, 2020, at 7:08 PM, 展荣臻（信泰） <zhanrzh_xt(a)teamsun.com.cn> wrote: > > > > Hi all, > > We have a hammer ceph cluster with 3 monitor,324 osds. OSD daemon and kvm is collocated on node; > > The ceph cluster are runing 2 years.Recently we added ~700 osds to the cluster,as process: > > 1.ceph osd create > > 2. mkdir -p /var/lib/ceph/osd/ceph-$osd > > 3. mkfs.xfs -f /dev/$disk > > 4. mount -o inode64,notime /dev/$disk /var/lib/ceph/osd/ceph-$osd > > 5. ceph-osd -i 0 --mkfs --mkkey > > 6.ceph auth add osd.$osd osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-$osd/keyring > > 7.ceph osd crush create-or-move $osd host=kvm101 root=default > > Mabe we do that requently.After add 122 osds, osd.1-osd.8 failed > > > > 2020-05-14 16:48:29.881021 7f6727fb9700 -1 common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f6727fb9700 time 2020-05-14 16:48:29.870051 > > common/Thread.cc: 129: FAILED assert(ret == 0) > > > > ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc8b55] > > 2: (Thread::create(unsigned long)+0x8a) [0xbac50a] > > 3: (Pipe::accept()+0x37fb) [0xca6c3b] > > 4: (Pipe::reader()+0x1a0f) [0xcaa75f] > > 5: (Pipe::Reader::entry()+0xd) [0xcb351d] > > 6: (()+0x7dc5) [0x7f67a45ebdc5] > > 7: (clone()+0x6d) [0x7f67a30cc1cd] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > ulimit -u > > 2061600 > > open files 32768 > > > > > > Does anyone know what's going on? Why create thread faild? > > > > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0