- ceph-users - lists.ceph.io

pubsub RGW and OSD processes suddenly start using much more CPU
by david.piper＠metaswitch.com 19 Aug '20

19 Aug '20

Hi all, I've got a 3-node cluster in a lab environment running on ceph version 14.2.9 (containerized). Each node is running a OSD, MON, MGR, MDS and 2 x RGW (we're using a second RGW instance to host a pubsub endpoint). I've been monitoring my nodes with the ceph dashboard, and noticed that all of a sudden at random the CPU profile of my hosts completely changes from [iowait: 20%, user: 10%, system: 5%, others: <5%] to [user: 40%, system: 20%, iowait: 5%, others: <5%]. This isn't a spike; the CPU profile is still like this after several days. All hosts seem to be affected at the same moment. Running `top` on the hosts shows it's the OSD and one of the RGW processes (seems to be the pubsub process) that are using all the CPU: $ top top - 14:57:23 up 11 days, 22:26, 1 user, load average: 1.65, 1.81, 2.04 Tasks: 140 total, 2 running, 138 sleeping, 0 stopped, 0 zombie %Cpu(s): 40.9 us, 21.4 sy, 0.0 ni, 27.9 id, 5.9 wa, 0.0 hi, 4.0 si, 0.0 st KiB Mem : 12304036 total, 183596 free, 10848600 used, 1271840 buff/cache KiB Swap: 0 total, 0 free, 0 used. 1184676 avail Mem Change delay from 3.0 to PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6541 167 20 0 5505324 205564 1380 S 67.8 1.7 4651:41 radosgw 3826 167 20 0 9563536 7.9g 5496 S 64.8 67.6 4609:05 ceph-osd 6621 167 20 0 5571764 276804 0 S 6.6 2.2 4985:30 radosgw 5524 167 20 0 1365368 322240 8516 S 3.7 2.6 171:37.23 ceph-mgr 1413 root 20 0 558728 28372 24968 S 0.3 0.2 18:56.05 rsyslogd .... If I restart the RGW instances (I think it's the pubsub instance specifically), CPU usage returns back to a normal level. I don't see any unusual logs (from the either RGW process or the OSD or MON processes) to indicate that anything has materially changed at the point the CPU elevates. Has anyone experienced anything like this? Beyond turning up RGW logging and trying to make sense of the output, I'm not sure where next to dig. Cheers, Dave

1 0

Convert existing rbd into a cinder volume
by Stolte, Felix 19 Aug '20

19 Aug '20

Hello fellow cephers, i know this is not the openstack mailing list, but since many of us are using ceph as a backend for openstack, maybe someone can help me out. I have a pool for rbds which are exported via iscsi to some bare metal windows servers and do snaphots on that rbds regulary. Now I want to clone a snapshot and attach the clone to an openstack instance (for restore purpose). I got it working, but the process is very uncomfortable and maybe someone can point me to a faster way. For now it looks like the following: 1. Create the clone via 'rbd clone <pool>/<image>@<snapshotname> <pool>/<clonename> 2. Import the clone into the openstack admin project with 'cinder manage --name=<my_volume_name> <cinderhost>@<pool>#<pool> 3. Create a volume transfer request in the admin project 4. Accept the transfer request in the desired project 5. Attach the volume to the instance There has to be an easier and faster way to accomplish this. Has someone a better solution? Best regards Felix ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

2 1

OSD takes more almost two hours to boot from Luminous -> Nautilus
by Mark Schouten 19 Aug '20

19 Aug '20

Hi, Last night I upgraded a Luminous cluster to Nautilus. All went well, but there was one sleep depriving issue I would like to prevent from happening next week while upgrading another cluster. Maybe you people can help me figure out what actually happened. So I upgraded the packages and restarted mons and mgrs. Then I started restarting the OSD's on one of the nodes. Below are the start and 'start_boot' times, during which the disks read at full speed, I think the whole disk. 2020-08-19 02:08:10.568 7fd742b09c80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:09:33.591 7fd742b09c80 1 osd.8 2188 start_boot 2020-08-19 02:08:10.592 7fb453887c80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:17:40.878 7fb453887c80 1 osd.5 2188 start_boot 2020-08-19 02:08:10.836 7f907bc0cc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:19:58.462 7f907bc0cc80 1 osd.3 2188 start_boot 2020-08-19 02:08:10.584 7f1ca892cc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 03:13:24.179 7f1ca892cc80 1 osd.11 2188 start_boot 2020-08-19 02:08:10.568 7f059f80dc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 04:06:55.342 7f059f80dc80 1 osd.14 2188 start_boot So, while this is not an issue which breaks anything technical, I would like to know how I can arrange for the OSD to do this 'maintenance' beforehand so I don't have to wait too long. :) I do see a warning in the logging, is that related: "store not yet converted to per-pool stats" ? Thanks! -- Mark Schouten <mark(a)tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208

2 3

How To Reset HP Printer? Get In Touch With HP Support Assistant.
by mary smith 19 Aug '20

19 Aug '20

If you are not aware of the process of getting the real time process of resetting your HP printer, then you should fetch the technical backing directly from the HP Support Assistant. Here, you will be able to get the top-to-toe solutions under the supervision of the experts in a couple of seconds. https://www.amiytech.com/hp-support-assistant/

1 0

What to Do When Printer Goes Down? Contact Epson Customer Service.
by mary smith 19 Aug '20

19 Aug '20

Found your printer is down? Printer issues tend to frustrate users at times, especially when you need it the most. Now, you not need to panic. Epson Customer Service professionals can diagnose the problems and fix your printer problem in a matter of minutes. You just need to share your problems, the error code and other errors. They avail expertise and trained exclusively to provide instant services. https://www.epsonprintersupportpro.net/

1 0

Alpine linux librados-dev missing
by Marc Roos 19 Aug '20

19 Aug '20

I am not sure if I should try this, but I was trying to build this dovecot-ceph-plugin plugin on alpine linux to create a nice small container image. However alpine linux does not seem to have librados-dev. Did any one do something similar, and have a workaround for this?

1 0

why ceph-fuse init Objecter with osd_timeout = 0
by Ch Wan 18 Aug '20

18 Aug '20

Hi all I'm using the mimic-13.2.4 version and ceph-fuse as client. Recently I'm suffering a strange problem: on client machine we can see the tcp has ESTAB state to osd machine, but on osd machine nothing could be found. Then the client hangs on read/write requests from this osd So I'm trying to figure out why this happens, and search for configuration to set a timeout for osd requests. I noticed that osdc/Objecter.h has a osd_timeout field, which will be initiated when we create an Objecter. But in ceph-fuse, it creates an Objecter using a fixed value 0, meaning no timeout? StandaloneClient::StandaloneClient(Messenger *m, MonClient *mc, > boost::asio::io_context& ictx) > : Client(m, mc, new Objecter(m->cct, m, mc, ictx, 0, 0)) > Here are my questions: 1. Why ceph-fuse set osd_timeout to 0? 2. Do we have other configurations to let osd requests fail instead of hanging forever?

1 0

fio rados ioengine
by Frank Ritchie 18 Aug '20

18 Aug '20

Hi all, When testing with the fio rados ioengine is it necessary to run a write test with the no-cleanup option before running read tests like is required with rados bench? thx Frank

1 0

Epson Printer Printing With Lines Contact Epson Customer Service.
by mary smith 18 Aug '20

18 Aug '20

You are printing structure Epson printer and getting lines on the print, to determine this issue contact Epson Customer Service. On the off chance that you are not happy with the data gave on the web, don't hesitate to get in touch with us. We are consistently there to keep you grinning. You can request help at any second as we are accessible all day, every day. Kindly don't stop for a second; we have a committed proficient to assist you with time. https://www.epsonprintersupportpro.net/

1 0

Re: New ceph cluster - cephx disabled, now without access
by Eugen Block 18 Aug '20

18 Aug '20

Have you tried restarting the ceph services? If those config options are not in ceph.conf and you removed them live they may reconnect after a restart. Or you put the config into ceph.conf and restart services and see if that helps to recover. Zitat von Tom Verhaeg <t.verhaeg(a)real-websolutions.nl>: > Hi, > > I just installed a new cluster with cephadm > (https://docs.ceph.com/docs/master/cephadm/install/) all was > working fine until I disabled cephx using the following commands: > > ceph config set global auth_cluster_required none > ceph config set global auth_service_required none > ceph config set global auth_client_required none > > Now I keep running into: > > [root@cph1]# ceph health > [errno 13] RADOS permission denied (error connecting to the cluster) > > However, in my /etc/ceph/ceph.conf there is no mention of the > configuration keys mentioned above. Same goes for the cluster folder > in: > > /var/lib/ceph/d5d206f4-8a4f-11ea-9d04-ac1f6bf8b2d3/mgr.cph1.srvnl.nl.qwnssx/config > > How can I remove this setting now? Obviously all ceph * related > commands run into the same RADOS permission denied. > > Thanks, > > Tom > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0