- ceph-users - lists.ceph.io

rgw print continue
by Seena Fallah 05 Jul '20

05 Jul '20

Hi all. I see this sentence in many sites. Does anyone knows why? > Then turn off print continue. If you have it set to true, you may encounter problems with PUT operations I use nginx in front of my rgw and proxy pass expect header in it. Thanks.

1 0

Placement of block/db and WAL on SSD?
by Lindsay Mathieson 05 Jul '20

05 Jul '20

Nautilus install. Documentation seems a bit ambiguous to me - this is for a spinner + SSD, using ceph-volume If I put the block.db on the SSD with "ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/sdc1" does the wal exists on the ssd (/dev/sdc1) as well, or does it remain on the hdd (/dev/sdd)? Conversely, what happens with the block.db if I place the wal with --block.wal Or do I have to setup separate partitions for the block.db and wal? -- Lindsay

2 3

High iops on bucket index
by Seena Fallah 05 Jul '20

05 Jul '20

Hi all. There are high iops on my bucket index pool when there is about 1K PUT request/s. Is there any way I can debug why there are so many iops on the bucket index pool? Thanks.

1 0

Showing OSD Disk config?
by Lindsay Mathieson 04 Jul '20

04 Jul '20

Is there a way to display an OSD's setup - data, data.db and WAL disks/partitions? -- Lindsay

3 3

Re: Ceph OSD not mounting after reboot
by Rodrigo Severo - Fábrica 04 Jul '20

04 Jul '20

Em sáb., 4 de jul. de 2020 às 11:27, Dave Hall <kdhall(a)binghamton.edu> escreveu: > > Rodrigo, > > I tried to send this to the list last night, but it looks like it didn't go through. I had a problem very much like this when I was first setting up my cluster. It turned out to be a missing systemd unit file. I would suggest that you check to see that you have an instance of ceph-volume@.service and an instance of ceph-osd@.service running for each OSD. I manually started my ceph-volume@ instances and then I managed to successfully restart my ceph-osd@ instances. My OSDs are back. Does anybody knows how is responsible for calling the ceph-volume@ instances on boot? Regards, Rodrigo

1 0

Re: Ceph OSD not mounting after reboot
by Rodrigo Severo - Fábrica 04 Jul '20

04 Jul '20

Em sáb., 4 de jul. de 2020 às 11:27, Dave Hall <kdhall(a)binghamton.edu> escreveu: > > Rodrigo, > > I tried to send this to the list last night, but it looks like it didn't go through. I had a problem very much like this when I was first setting up my cluster. It turned out to be a missing systemd unit file. I would suggest that you check to see that you have an instance of ceph-volume@.service and an instance of ceph-osd@.service running for each OSD. > > ceph-volume(a)lvm-10-a63a2465-9de2-497e-bf60-72ed4c3c4c33.service > ceph-volume(a)lvm-11-a2151f1f-84f5-407b-a730-9d2a9502a85f.service > ceph-volume(a)lvm-12-7a16123b-7a0f-40a2-9159-5555724d9978.service > ceph-volume(a)lvm-13-bda32c07-ecc7-40c4-8066-383977c5e795.service > ceph-volume(a)lvm-14-4cc89d3a-5390-415d-b12b-9a557b7b5950.service > ceph-volume(a)lvm-15-21bbdaf5-1994-43a9-aa80-982689f51438.service > ceph-volume(a)lvm-8-cda4394b-e132-4530-8044-3cbbfbcbea19.service > ceph-volume(a)lvm-9-72ff199d-a4e4-4d26-9a4f-00337f8fdc7c.service > > ceph-osd(a)10.service > ceph-osd(a)12.service > ceph-osd(a)14.service > ceph-osd(a)8.service > ceph-osd(a)11.service > ceph-osd(a)13.service > ceph-osd(a)15.service > ceph-osd(a)9.service > > In my case I found that the proto-unit for one of these (ceph-volume@.service, I think) was missing from /var/lib/systemd/system. In fact, it was missing from the Debian install package for some reason. I think I had to unpack a copy of the Debian SRC pacakge to retrieve the file. Once I added the missing file and made sure the unit was enabled, things started working better and making more sense. I don't recall if all of the instances created themselves or whether I had to do something addtional, but it worked. Hi Dave, I'm looking around but can't identify any missing file. ceph-osd@.service and ceph-volume@.service are present. I'm looking on the other servers for some other file that might be missing but can't find anything. Thanks for your help, Rodrigo

1 0

Re: Ceph OSD not mounting after reboot
by Rodrigo Severo - Fábrica 04 Jul '20

04 Jul '20

Em sex., 3 de jul. de 2020 às 17:41, Marc Roos <M.Roos(a)f1-outsourcing.eu> escreveu: > > So mount it, if it is empty Sure. That was my first impulse but as I said, in my other osd servers, these mounts are tmpfs filesystems. It's easy to manually mount them but how would I populate them? Regards, Rodrigo Severo > > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] Ceph OSD not mounting after reboot > > Hi, > > > Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to > 14.2.10 and it's OSDs won't come up. > > I find the following messages on my log: > > 4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 > 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on > /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory > 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 > 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring > found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx > > and my /var/lib/ceph/osd/ceph-6 directory is empty. > > I see that on my other servers these /var/lib/ceph/osd/ceph-? > directories are tmpfs mounts but I can't understand who is responsible > for mounting them as there are no entries for them in /etc/fstab. > > How can I fix this osd server? > > > Regards, > > Rodrigo Severo > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

2 1

Ceph OSD not mounting after reboot
by Rodrigo Severo - Fábrica 04 Jul '20

04 Jul '20

Hi, Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to 14.2.10 and it's OSDs won't come up. I find the following messages on my log: 4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx and my /var/lib/ceph/osd/ceph-6 directory is empty. I see that on my other servers these /var/lib/ceph/osd/ceph-? directories are tmpfs mounts but I can't understand who is responsible for mounting them as there are no entries for them in /etc/fstab. How can I fix this osd server? Regards, Rodrigo Severo

4 4

[Octopus] OSD won’t work with Docker
by Sean Johnson 03 Jul '20

03 Jul '20

I have a situation were OSDs won’t work as Docker containers with Octopus on an Ubuntu 2020.04 host. The cephadm adopt —style legacy —name osd.8 command works as expected, and sets up the /var/lib/ceph/ directory as expected: root@balin:~# ll /var/lib/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667/osd.8/ total 64 drwx------ 2 167 167 4096 Jul 3 08:39 ./ drwx------ 8 167 167 4096 Jul 3 08:30 ../ lrwxrwxrwx 1 167 167 93 Jul 3 08:39 block -> /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-block-5b229640-14e5-4e7f-9993-9368d172c30c -rw------- 1 167 167 37 Jul 3 08:39 ceph_fsid -rw------- 1 167 167 283 Jul 3 08:14 config -rw------- 1 167 167 37 Jul 3 08:39 fsid -rw------- 1 167 167 55 Jul 3 08:39 keyring -rw------- 1 167 167 6 Jul 3 08:39 ready -rw------- 1 167 167 3 May 14 21:41 require_osd_release -rw------- 1 167 167 10 Jul 3 08:39 type -rw------- 1 167 167 38 Jul 3 08:14 unit.configured -rw------- 1 167 167 48 May 14 21:41 unit.created -rw------- 1 167 167 28 Jul 3 08:14 unit.image -rw------- 1 167 167 825 Jul 3 08:14 unit.poststop -rw------- 1 167 167 2367 Jul 3 08:14 unit.run -rw------- 1 167 167 2 Jul 3 08:39 whoami However, the problem shows up when the container starts. Jul 03 08:39:40 balin systemd[1]: Started Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-blo> Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/ln -snf /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-block-5b229640-14e5-4e7f-9993-9368d172c30c /var/li> Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-8/block Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 Jul 03 08:39:41 balin bash[5412]: --> ceph-volume lvm activate successful for osd ID: 8 Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0 set uid:gid to 167:167 (ceph:ceph) Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable), pro> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0 pidfile_write: ignore empty --pid-file Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open size 12000134430720 (0xae9f> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bluestore(/var/lib/ceph/osd/ceph-8) _set_cache_sizes cache_size 1073741824 meta 0.4 > Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) open size 12000134430720 (0xae9f> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-8/block size 11 TiB Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) close Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.888+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) close Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.141+0000 7f52b1e70f40 0 starting osd.8 osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 0 load: jerasure load: lrc load: isa Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 -1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open open got: (13) Permission d> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/c> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1 bluestore(/var/lib/ceph/osd/ceph-8) _mount path /var/lib/ceph/osd/ceph-8 Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/c> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open open got: (13) Permission d> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 osd.8 0 OSD:init: unable to mount object store Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 ** ERROR: osd init failed: (13) Permission denied Jul 03 08:39:42 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Main process exited, code=exited, status=1/FAILURE Jul 03 08:39:42 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Failed with result 'exit-code'. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Scheduled restart job, restart counter is at 6. Jul 03 08:39:53 balin systemd[1]: Stopped Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Start request repeated too quickly. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Failed with result 'exit-code'. Jul 03 08:39:53 balin systemd[1]: Failed to start Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. There’s a permission denied issue with the block, and I can’t find what might be causing that. If I set up an OSD fresh, it works fine with the docker container until I reboot, and then I get the same permission denied error. If I use ceph-volume lvm activate a standard ceph systemd service is created, and the OSD comes online without any problem, though it’s not visible to ceph orch.

1 1

Object Gateway not working within the dashboard anymore after network change
by Hendrik Peyerl 03 Jul '20

03 Jul '20

Hi all, we are currently experiencing a problem with the Obejct Gateway part of the dashboard not working anymore: We had a working setup were the RGW servers only had 1 network interface with an IP address that was reachable by the monitor servers and the dashboard was working as expected. After our initial tests everything was working great and we decided to add another physical link to the RGW Servers for the traffic to the clients. With that network change we also had to set the default gateway to the new interface while adding static routes for the rest of the ceph environment. To avoid issues with hostnames (the old hostname now resolves to the new interface) we added another hostname for the internal traffic, purged the gateways from ceph and added them again via ceph-deploy rgw create with the new hostname. The S3 communication is working perfectly fine as it did before, we can reach all buckets and the monitors can communicate with the Gateway. The Dashboard however throws the following error whenever we navigate to any of the object gateway menus: ————————————————————————————————— 2020-07-03 10:33:41.871 7fa0f9dbc700 0 mgr[dashboard] [03/Jul/2020:10:33:41] HTTP Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond response.body = self.handler() File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap return self.newhandler(innerfunc, *args, **kwargs) File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler return handler(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 661, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 28, in status if not instance.is_service_online(): File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 507, in func_wrapper **kwargs) File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 321, in is_service_online _ = request({'format': 'json'}) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 313, in __call__ data, raw_content) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 445, in do_request ex.args[0].reason.args[0]) File "/usr/lib64/python2.7/re.py", line 137, in match return _compile(pattern, flags).match(string) TypeError: expected string or buffer 2020-07-03 10:33:41.872 7fa0f9dbc700 0 mgr[dashboard] [2a02:2e0:13::a05:42784] [GET] [500] [45.044s] [plusline] [1.8K] /api/rgw/status 2020-07-03 10:33:41.872 7fa0f9dbc700 0 mgr[dashboard] ['{"status": "500 Internal Server Error", "version": "3.2.2", "traceback": "Traceback (most recent call last):\\n File \\"/usr/lib/python2.7/site- packages/cherrypy/_cprequest.py\\", line 656, in respond\\n response.body = self.handler()\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py\\", line 188, in __call__\\n self.b ody = self.oldhandler(*args, **kwargs)\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/_cptools.py\\", line 221, in wrap\\n return self.newhandler(innerfunc, *args, **kwargs)\\n File \\"/usr/s hare/ceph/mgr/dashboard/services/exception.py\\", line 88, in dashboard_exception_handler\\n return handler(*args, **kwargs)\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py\\", l ine 34, in __call__\\n return self.callable(*self.args, **self.kwargs)\\n File \\"/usr/share/ceph/mgr/dashboard/controllers/__init__.py\\", line 661, in inner\\n ret = func(*args, **kwargs)\\n F ile \\"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\\", line 28, in status\\n if not instance.is_service_online():\\n File \\"/usr/share/ceph/mgr/dashboard/rest_client.py\\", line 507, in func_w rapper\\n **kwargs)\\n File \\"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\\", line 321, in is_service_online\\n _ = request({\'format\': \'json\'})\\n File \\"/usr/share/ceph/mgr/dashb oard/rest_client.py\\", line 313, in __call__\\n data, raw_content)\\n File \\"/usr/share/ceph/mgr/dashboard/rest_client.py\\", line 445, in do_request\\n ex.args[0].reason.args[0])\\n File \\"/usr/lib64/python2.7/re.py\\", line 137, in match\\n return _compile(pattern, flags).match(string)\\nTypeError: expected string or buffer\\n", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "e0d6ff11-4dad-496a-9ee7-9db036c46ab7"}'] ————————————————————————————————— We are running ceph version 14.2.9 on CentOS 7.7. Any help on how to debug this would be greatly apreciated. Best Regards, Hendrik

2 3