September 2019 - ceph-users

Re: FileStore OSD, journal direct symlinked, permission troubles.
by Marco Gaiarin 11 Sep '19

11 Sep '19

Riprendo quanto scritto nel suo messaggio del 29/08/2019... > Another possibilty is to convert the MBR to GPT (sgdisk --mbrtogpt) and > give the partition its UID (also sgdisk). Then it could be linked by > its uuid. and, in another email: > And I forgot that you can also re-create the journal by itself. I can't > recall the command ATM though. Ahem, i stated the jornal disk are also the OS disks, and i'm using old server, so i think that converting to GPT will lead to an unbootable node... But, the 'code' that identify (and change permission) for journal dev are PVE specific? or Ceph generic? I suppose the latter... Also, i've done: adduser ceph disk and partition devices are '660 root:disk': why still i get 'permission denied'? > Or if you are not in need of filestore OSDs, re-create them as bluestore > ones. AFAICS, Ceph has laid more focus on bluestore and it might be > better to do a conversion sooner than later. (my opinion) Not for now; bluestore migration need a bit more time/study/knowledge... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

2 6

Warning: 1 pool nearfull and unbalanced data distribution
by Thomas 11 Sep '19

11 Sep '19

Hi, the output of ceph health details gives me a warning that concerns me a little. I'll explain in a second. root@ld3955:/mnt/rbd# ceph health detail HEALTH_WARN 1 nearfull osd(s); 1 pool(s) nearfull; 4 pools have too many placement groups OSD_NEARFULL 1 nearfull osd(s) osd.122 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'hdb_backup' is nearfull POOL_TOO_MANY_PGS 4 pools have too many placement groups Pool pve_cephfs_data has 128 placement groups, should have 16 Pool hdd has 512 placement groups, should have 64 Pool pve_cephfs_metadata has 32 placement groups, should have 4 Pool backup has 1024 placement groups, should have 4 I'm writing +90% of the data in pool "hdb_backup", and this is ongoing. Therefore I can hardly afford that this pool is full. When I check the output of ceph osd status the relevand osd is somehow overutilized: root@ld3955:/mnt/rbd# ceph osd status | grep nearfull +-----+--------+-------+-------+--------+---------+--------+---------+--------------------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +-----+--------+-------+-------+--------+---------+--------+---------+--------------------+ | 122 | ld5505 | 1448G | 227G | 0 | 0 | 0 | 0 | exists,nearfull,up | This looks like an inconsistency, but I can check the usage with ceph osd df. Here I can see that the OSDs are not really balanced: root@ld3955:/mnt/rbd# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 272 hdd 7.28000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 136 KiB 3.7 GiB 5.0 TiB 31.23 0.77 121 up 273 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 28 KiB 5.1 GiB 4.4 TiB 39.35 0.97 152 up 274 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 168 KiB 4.5 GiB 4.4 TiB 39.41 0.97 152 up 275 hdd 7.28000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 139 KiB 3.5 GiB 5.1 TiB 29.77 0.73 115 up 276 hdd 7.28000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 48 KiB 5.7 GiB 4.5 TiB 38.81 0.96 150 up 277 hdd 7.28000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 276 KiB 4.3 GiB 4.8 TiB 34.68 0.85 134 up 278 hdd 7.28000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 KiB 4.4 GiB 4.5 TiB 38.74 0.95 150 up 279 hdd 7.28000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 156 KiB 4.1 GiB 4.7 TiB 35.80 0.88 138 up 280 hdd 7.28000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 156 KiB 4.3 GiB 4.6 TiB 37.03 0.91 143 up 281 hdd 7.28000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 172 KiB 3.8 GiB 4.9 TiB 32.67 0.80 126 up 282 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 120 KiB 4.5 GiB 4.4 TiB 39.39 0.97 152 up 283 hdd 7.28000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 32 KiB 5.9 GiB 4.5 TiB 37.57 0.93 145 up [...] 76 hdd 1.64000 1.00000 1.6 TiB 1.4 TiB 1.4 TiB 88 KiB 2.4 GiB 268 GiB 84.02 2.07 73 up 77 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 188 KiB 2.0 GiB 560 GiB 66.59 1.64 58 up 78 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1023 GiB 164 KiB 1.9 GiB 651 GiB 61.15 1.51 53 up 79 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 176 KiB 1.9 GiB 636 GiB 62.02 1.53 54 up 80 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 80 KiB 2.5 GiB 636 GiB 62.07 1.53 54 up 81 hdd 1.64000 1.00000 1.6 TiB 886 GiB 885 GiB 128 KiB 1.7 GiB 790 GiB 52.89 1.30 46 up 82 hdd 1.64000 1.00000 1.6 TiB 967 GiB 965 GiB 240 KiB 1.8 GiB 709 GiB 57.70 1.42 50 up 83 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.2 GiB 420 GiB 74.94 1.85 65 up 84 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.0 GiB 597 GiB 64.37 1.59 56 up 85 hdd 1.64000 1.00000 1.6 TiB 811 GiB 810 GiB 176 KiB 1.6 GiB 865 GiB 48.42 1.19 42 up 86 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB 613 GiB 63.43 1.56 55 up 87 hdd 1.64000 1.00000 1.6 TiB 791 GiB 789 GiB 68 KiB 1.6 GiB 885 GiB 47.17 1.16 41 up 88 hdd 1.64000 1.00000 1.6 TiB 908 GiB 906 GiB 168 KiB 1.8 GiB 768 GiB 54.18 1.33 47 up [...] 113 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 100 KiB 3.0 GiB 342 GiB 79.60 1.96 69 up 114 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 999 GiB 184 KiB 1.9 GiB 675 GiB 59.70 1.47 52 up 115 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB 407 GiB 75.70 1.86 66 up 116 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 92 KiB 2.0 GiB 597 GiB 64.39 1.59 56 up 117 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 76 KiB 2.7 GiB 480 GiB 71.34 1.76 62 up 118 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.6 GiB 574 GiB 65.74 1.62 57 up 119 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 152 KiB 1.9 GiB 634 GiB 62.19 1.53 54 up 120 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.0 GiB 541 GiB 67.73 1.67 59 up 121 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.0 GiB 556 GiB 66.82 1.65 58 up 122 hdd 1.64000 0.95001 1.6 TiB 1.4 TiB 1.4 TiB 184 KiB 2.5 GiB 227 GiB 86.44 2.13 75 up I assume that this is a result of my Ceph Cluster history, means I started with 4 OSD nodes with 48 drives @1.8TB. This was 345TB in total. I started to fill this storage up to 75%. Then I added 2 OSD nodes with 48 drives @8TB. I was not expecting that pool "hdb_backup" would be filled up in the near future. This is the ouput of ceph df: root@ld3955:/mnt/rbd# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.1 PiB 661 TiB 467 TiB 468 TiB 41.43 nvme 23 TiB 23 TiB 681 MiB 8.7 GiB 0.04 TOTAL 1.1 PiB 685 TiB 467 TiB 468 TiB 40.59 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL backup 4 0 B 0 0 B 0 35 TiB hdb_backup 11 154 TiB 40.39M 154 TiB 64.18 29 TiB hdd 30 1.1 TiB 281.21k 1.1 TiB 1.01 35 TiB pve_cephfs_data 32 318 GiB 91.83k 318 GiB 0.30 35 TiB pve_cephfs_metadata 33 155 MiB 61 155 MiB 0 35 TiB nvme 35 0 B 0 0 B 0 7.4 TiB Question: How can I start rebalancing data in order to have more data in the larger drives (8TB)? Or is it ok that the smaller drives (1.8TB) are filled by +60%? THX for your advice Thomas

1 0

unsubscribe
by Gökhan Kocak 11 Sep '19

11 Sep '19

2 1

2 OpenStack environment, 1 Ceph cluster
by vladimir franciz blando 11 Sep '19

11 Sep '19

I have 2 OpenStack environment that I want to integrate to an existing ceph cluster. I know technically it can be done but has anyone tried this? - Vlad ᐧ

5 5

[nautilus] Dashboard & RADOSGW
by DHilsbos＠performair.com 10 Sep '19

10 Sep '19

All; We're trying to add a RADOSGW instance to our new production cluster, and it's not showing in the dashboard, or in ceph -s. The cluster is running 14.2.2, and the RADOSGW got 14.2.3. systemctl status ceph-radosgw@ rgw.s700037 returns: active (running). ss -ntlp does NOT show port 80. Here's the ceph.conf on the system: [global] fsid = effc5134-e0cc-4628-a079-d67b60071f90 mon initial members = s700034,s700035,s700036 mon host = [v1:10.0.80.10:6789/0,v2:10.0.80.10:3300/0],[v1:10.0.80.11:6789/0,v2:10.0.80.11:3300/0],[v1:10.0.80.12:6789/0,v2:10.0.80.12:3300/0] public network = 10.0.80.0/24 cluster network = 10.0.88.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 8 osd pool default pgp num = 8 [client.rgw.s700037] host = s700037.performair.local rgw frontends = "civetweb port=80" rgw dns name = radosgw.performair.local Any thoughts on what I'm missing? I'm also seeing these in the manager's logs: 2019-09-10 15:49:43.946 7efe6eee1700 0 mgr[dashboard] [10/Sep/2019:15:49:43] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1837, in start self.tick() File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1902, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/ssl_builtin.py", line 52, in wrap keyfile=self.private_key, ssl_version=ssl.PROTOCOL_SSLv23) File "/usr/lib64/python2.7/ssl.py", line 934, in wrap_socket ciphers=ciphers) File "/usr/lib64/python2.7/ssl.py", line 609, in __init__ self.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:618) Thoughts on this? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

1 1

Manager plugins issues on new ceph-mgr nodes
by Alexandru Cucu 10 Sep '19

10 Sep '19

Hello, Running 14.2.3, updated from 14.2.1. Until recently I've had ceph-mgr collocated with OSDs. I've installed ceph-mgr on separate servers and everything looks OK in Ceph status but there are multiple issues: 1. Dashboard only runs on old mgr servers. Tried restarting the daemons and disable/enable the dashboard plugin. New mgr won't listen on the dashboard port. 2. To (re)enable the dashboard plugin I had to use "--force" # ceph mgr module enable dashboard Error ENOENT: all mgr daemons do not support module 'dashboard', pass --force to force enablement 3. When accessing the Cluster -> Manager modules menu in the dashboard I get a 500 error message. The exact error below: ---- 2019-09-10 15:01:39.270 7fb6d4916700 0 mgr[dashboard] [10/Sep/2019:15:01:39] HTTP Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond response.body = self.handler() File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap return self.newhandler(innerfunc, *args, **kwargs) File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler return handler(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 842, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/mgr_modules.py", line 35, in list obj['enabled'] = True TypeError: 'NoneType' object does not support item assignment 2019-09-10 15:01:39.271 7fb6d4916700 0 mgr[dashboard] [::ffff:192.168.15.55:54860] [GET] [500] [0.014s] [admin] [1.3K] /api/mgr/module 2019-09-10 15:01:39.272 7fb6d4916700 0 mgr[dashboard] ['{"status": "500 Internal Server Error", "version": "3.2.2", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "traceback": "Traceback (most recent call last):\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py\\", line 656, in respond\\n response.body = self.handler()\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py\\", line 188, in __call__\\n self.body = self.oldhandler(*args, **kwargs)\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/_cptools.py\\", line 221, in wrap\\n return self.newhandler(innerfunc, *args, **kwargs)\\n File \\"/usr/share/ceph/mgr/dashboard/services/exception.py\\", line 88, in dashboard_exception_handler\\n return handler(*args, **kwargs)\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py\\", line 34, in __call__\\n return self.callable(*self.args, **self.kwargs)\\n File \\"/usr/share/ceph/mgr/dashboard/controllers/__init__.py\\", line 649, in inner\\n ret = func(*args, **kwargs)\\n File \\"/usr/share/ceph/mgr/dashboard/controllers/__init__.py\\", line 842, in wrapper\\n return func(*vpath, **params)\\n File \\"/usr/share/ceph/mgr/dashboard/controllers/mgr_modules.py\\", line 35, in list\\n obj[\'enabled\'] = True\\nTypeError: \'NoneType\' object does not support item assignment\\n"}'] ---- Anyone got the same problems after adding new manager nodes? Is there something I'm missing here? Thanks! --- Alex Cucu

2 1

Host failure trigger " Cannot allocate memory"
by Amudhan P 10 Sep '19

10 Sep '19

Hi, I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. My current setup: 3 nodes, 1 node contain two bricks and other 2 nodes contain single brick each. Volume is a 3 replica, I am trying to simulate node failure. I powered down one host and started getting msg in other systems when running any command "-bash: fork: Cannot allocate memory" and system not responding to commands. what could be the reason for this? at this stage, I could able to read some of the data stored in the volume and some just waiting for IO. output from "sudo ceph -s" cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN 1 osds down 2 hosts (3 osds) down Degraded data redundancy: 5313488/7970232 objects degraded (66.667%), 64 pgs degraded services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 4 osds: 1 up, 2 in data: pools: 2 pools, 64 pgs objects: 2.66 M objects, 206 GiB usage: 421 GiB used, 3.2 TiB / 3.6 TiB avail pgs: 5313488/7970232 objects degraded (66.667%) 64 active+undersized+degraded io: client: 79 MiB/s rd, 24 op/s rd, 0 op/s wr output from : sudo ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 0 hdd 1.81940 0 0 B 0 B 0 B 0 0 0 3 hdd 1.81940 0 0 B 0 B 0 B 0 0 0 1 hdd 1.81940 1.00000 1.8 TiB 211 GiB 1.6 TiB 11.34 1.00 0 2 hdd 1.81940 1.00000 1.8 TiB 210 GiB 1.6 TiB 11.28 1.00 64 TOTAL 3.6 TiB 421 GiB 3.2 TiB 11.31 MIN/MAX VAR: 1.00/1.00 STDDEV: 0.03 regards Amudhan

2 4

(no subject)
by Amudhan P 10 Sep '19

10 Sep '19

Hi, I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. My current setup: 3 nodes, 1 node contain two bricks and other 2 nodes contain single brick each. Volume is a 3 replica, I am trying to simulate node failure. I powered down one host and started getting msg in other systems when running any command "-bash: fork: Cannot allocate memory" and system not responding to commands. what could be the reason for this? at this stage, I could able to read some of the data stored in the volume and some just waiting for IO. output from "sudo ceph -s" cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN 1 osds down 2 hosts (3 osds) down Degraded data redundancy: 5313488/7970232 objects degraded (66.667%), 64 pgs degraded services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 4 osds: 1 up, 2 in data: pools: 2 pools, 64 pgs objects: 2.66 M objects, 206 GiB usage: 421 GiB used, 3.2 TiB / 3.6 TiB avail pgs: 5313488/7970232 objects degraded (66.667%) 64 active+undersized+degraded io: client: 79 MiB/s rd, 24 op/s rd, 0 op/s wr output from : sudo ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 0 hdd 1.81940 0 0 B 0 B 0 B 0 0 0 3 hdd 1.81940 0 0 B 0 B 0 B 0 0 0 1 hdd 1.81940 1.00000 1.8 TiB 211 GiB 1.6 TiB 11.34 1.00 0 2 hdd 1.81940 1.00000 1.8 TiB 210 GiB 1.6 TiB 11.28 1.00 64 TOTAL 3.6 TiB 421 GiB 3.2 TiB 11.31 MIN/MAX VAR: 1.00/1.00 STDDEV: 0.03 regards Amudhan

1 0

ceph -openstack -kolla-ansible deployed using docker containers - One OSD is down out of 4- how can I bringt it up
by Reddi Prasad Yendluri 10 Sep '19

10 Sep '19

Hi Team, I have a production Openstack which was deployed using kolla-ansible and docker containerization has been used to deploy Ceph cluster storage. Now, 1/4 OSD is down. How can I bring it up. Basically, management of docker containerized application and restart the services or bring up the OSD services on one of the Ceph node. Can you please support me if any of you aware of this? Thanks, Reddi Prasad YENDLURI Cloud Specialist M +65 8345 9599 | D +65 6220 9908 Office: *51B Circular Road Singapore 049406* [image: PALO IT]

1 1

Unable to replace OSDs deployed with ceph-volume lvm batch
by Burkhard Linke 09 Sep '19

09 Sep '19

Hi, we had a failing hard disk, and I replace it and want to create a new OSD on it now. But ceph-volume fails under these circumstances. In the original setup, the OSDs were created with ceph-volume lvm batch using a bunch of drives and a NVMe device for bluestore db. The batch mode uses a volume group on the NVMe device instead of partitions.I have removed the former db logical volume, the lvm setup for the former hard disk and all other remainders. Creating a new OSD with any combination of devices now fails: --data /dev/sda --block.db <nvme device> --data /dev/sda --block.db <nvme volume group> --data /dev/sda --block.db <lv created manually in nvme volume group> # ceph-volume lvm create --bluestore --data /dev/sda --block.db /dev/ceph-block-dbs-ea684aa8-544e-4c4a-8664-6cb50b3116b8/osd-block-db-a8f1489a-d97b-479e-b9a7-30fc9fa99cb5 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 55cfb9f8-aa30-4f8b-8b95-a43d3f97fe5b Running command: /sbin/vgcreate -s 1G --force --yes ceph-1a2ddc14-780b-45e4-a036-e928862e6ccb /dev/sda stdout: Physical volume "/dev/sda" successfully created. stdout: Volume group "ceph-1a2ddc14-780b-45e4-a036-e928862e6ccb" successfully created Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-55cfb9f8-aa30-4f8b-8b95-a43d3f97fe5b ceph-1a2ddc14-780b-45e4-a036-e928862e6ccb stdout: Logical volume "osd-block-55cfb9f8-aa30-4f8b-8b95-a43d3f97fe5b" created. --> blkid could not detect a PARTUUID for device: /dev/ceph-block-dbs-ea684aa8-544e-4c4a-8664-6cb50b3116b8/osd-block-db-a8f1489a-d97b-479e-b9a7-30fc9fa99cb5 --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.136 --yes-i-really-mean-it stderr: purged osd.136 --> RuntimeError: unable to use device In all cases ceph-volume is not able to detect a partition uuid for the db device (which is correct, since the device is a logical volume....). Running 'ceph-volume lvm batch' again results in a OSD without using the NVMe device as db. So what is the recommended way to manually create an OSD with a certain hard disk and an existing logical volume as db partition? I would like to avoid to zap all other OSDs using the NVMe device and recreate them in a single run with 'ceph-volume lvm batch ...'. Regards, Burkhard

2 2

ceph-iscsi and tcmu-runner RPMs for CentOS?
by Robert Sander 09 Sep '19

09 Sep '19

Hi, In the Documentation on https://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli/ it is stated that you need at least CentOS 7.5 with at least kernel 4.16 and to install tcmu-runner and ceph-iscsi "from your Linux distribution's software repository". CentOS does not know about tcmu-runner nor ceph-iscsi. Where do I get these RPMs from? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

Bucket policies with OpenStack integration and limiting access
by shubjero＠gmail.com 09 Sep '19

09 Sep '19

Good day, We have a Ceph cluster and make use of object-storage and integrate with OpenStack. Each OpenStack project/tenant is given a radosgw user which allows all keystone users of that project to access the object-storage as that single radosgw user. The radosgw user is the project id of the OpenStack project/tenant. Sometimes we have use cases where we want to access the object-storage outside of the swift-api and use tools like the aws-cli or homebrew java applications to access the object storage. For this use case what we do is generate S3 access/secret key for the specific radosgw user and they have full access to the object storage for that OpenStack project/tenant. What we want to know is if it is possible to provide granular access to containers within a single OpenStack project using S3 access keys or S3 sub-users? I know that the Swift API has ACL's that can limit by keystone user but we are exploring the possibility of doing this using S3 and S3 bucket policies so that the tools our team are developing (open source) are more transferrable to AWS S3 and Rados GW. Thanks all, Jared Baker Cloud Architect, OICR

1 0

Out of memory
by Sylvain PORTIER 09 Sep '19

09 Sep '19

Hi, On my ceph osd servers I have lot of "out of memory messages". My servers are configured with : - 32 G of memory - 11 HDD (3,5 T each) (+ 2 HDD for the system) And the error messages are : /[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child// //[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB// / Top result : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd Is there a way to limit or reduce the memory usage of each osd deamon ? Thank you, Regards, Sylvain PORTIER. --- L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast. https://www.avast.com/antivirus

4 5

How to test PG mapping with reweight
by Robert LeBlanc 07 Sep '19

07 Sep '19

I'd like to test how reweighting an OSD will change how the PGs map in the cluster. I suspect that I'd dump the CRUSH map and PGs in the cluster that I'm interested in then use osdmaptool. I'm not understanding how to use osdmaptool to set the reweight, then query a PG or the entire set of PGs that I'm interested in. I then suspect that if I'm okay with the new map that I could inject it into the cluster instead of having to run reweight on the OSD(s). This is a Jewel cluster and I'm trying to calculate OSD usage offline, then inject a map that is more distributed instead of doing a reweight, move the PGs which take a long time to just rinse and repeat over and over again. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 3

Ceph client failed to mount RBD device after reboot
by Vang Le-Quy 06 Sep '19

06 Sep '19

I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client n01 and n03 with following software versions. # n01 ceph-common: Installed: 12.2.8-0ubuntu0.18.04.1 Candidate: 12.2.12-0ubuntu0.18.04.2 # n03 ceph-common: Installed: 12.2.11-0ubuntu0.18.04.2 Candidate: 12.2.12-0ubuntu0.18.04.2 # n01 uname -a Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # n03 uname -a Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Problem: N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on client side, I upgrade n01 to have the same package version to n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the following: Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected [signature_1482564062] Vang Quy Le Special Consultant in Data Science and Infrastructure T: (+45) 9940 7710 | Email: vle(a)its.aau.dk<mailto:vle@its.aau.dk> Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |

4 5

Re: using non client.admin user for ceph-iscsi gateways
by Jason Dillaman 06 Sep '19

06 Sep '19

On Fri, Sep 6, 2019 at 12:00 PM Wesley Dillingham <wdillingham(a)godaddy.com> wrote: > > the iscsi-gateway.cfg seemingly allows for an alternative cephx user other than client.admin to be used, however the comments in the documentations says specifically to use client.admin. Hmm, can you point out where this is in the docs? Originally, tcmu-runner didn't support the ability to change the user id, but that has been available for about a year now [1]. > Other than having the cfg file point to the appropriate key/user with "gateway_keyring" and giving that client read caps on the mons and full access to the pool configured to be used for iscsi are any other particular steps / settings / actions needed? Just use "profile rbd" for your caps to keep it simple. > It seems prudent to not use client.admin but I don't want to have unstable behavior or untested setup. > > Thanks. > > Respectfully, > > Wes Dillingham > wdillingham(a)godaddy.com > Site Reliability Engineer IV - Platform Storage / Ceph > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io [1] https://github.com/open-iscsi/tcmu-runner/commit/c85ccdcfb7f4b17926eda1df89… -- Jason

2 1

using non client.admin user for ceph-iscsi gateways
by Wesley Dillingham 06 Sep '19

06 Sep '19

the iscsi-gateway.cfg seemingly allows for an alternative cephx user other than client.admin to be used, however the comments in the documentations says specifically to use client.admin. Other than having the cfg file point to the appropriate key/user with "gateway_keyring" and giving that client read caps on the mons and full access to the pool configured to be used for iscsi are any other particular steps / settings / actions needed? It seems prudent to not use client.admin but I don't want to have unstable behavior or untested setup. Thanks. Respectfully, Wes Dillingham wdillingham(a)godaddy.com Site Reliability Engineer IV - Platform Storage / Ceph

1 0

Followup: weird behaviour with ceph osd pool create and the "crush-rule" parameter (suddenly changes behaviour)
by aoanla＠gmail.com 06 Sep '19

06 Sep '19

So, whilst debugging the behaviour in the first thread I created, I needed to create and then destroy pools (to avoid running out of placement groups). So, I did something like: ceph osd pool create ec2pool 2048 2048 erasure glasgow-eci-test ec2pool 0 ceph osd pool create ec3pool 2048 2048 erasure glasgow-eci-test2 ec3pool 0 (for two different types of ecpool) and then removed them with ceph osd pool rm ec2pool ec2pool --yes-i-really-really-mean-it ceph osd pool rm ec3pool ec3pool --yes-i-really-really-mean-it Now, however, something seems to have broken, as if I attempt: ceph osd pool create ec4pool 2048 2048 erasure glasgow-eci-test3 ec4pool 0 it fails with Error ENOENT: specified rule ec4pool doesn't exist (which, of course, it does not, as the whole point of the syntax is that ceph should build the crush rule for me and name it appropriately; and this worked for all the previous times). ceph health returns HEALTH OK still. Any suggestions? I've googled around a bit on this, but I can't seem to find anyone discussing it... Sam

4 6

RGW bucket check --check-objects -fix failed
by EDH - Manuel Rios Fernandez 06 Sep '19

06 Sep '19

Hi, Were at 14.2.2 We just found a broken bucket index, trying to repair with the common commands ]# radosgw-admin bucket check --check-objects fix finish instantly, but bucket should have near 60-70TB info. [root@CEPH-MON01 home]# radosgw-admin bucket check --check-objects --bucket BUCKETNAME --debug_rgw=10 2019-09-05 20:08:13.908 7fdc8db37580 10 cannot find current period zonegroup using local zonegroup 2019-09-05 20:08:13.910 7fdc8db37580 10 Cannot find current period zone using local zone 2019-09-05 20:08:13.933 7fdc8db37580 2 all 8 watchers are set, enabling cache 2019-09-05 20:08:13.942 7fdc56ffd700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-09-05 20:08:13.948 7fdc8db37580 10 cache get: name=default.rgw.data.root++BUCKETNAME : miss 2019-09-05 20:08:13.950 7fdc8db37580 10 cache put: name=default.rgw.data.root++BUCKETNAME info.flags=0x16 2019-09-05 20:08:13.950 7fdc8db37580 10 adding default.rgw.data.root++BUCKETNAME to cache LRU end 2019-09-05 20:08:13.950 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.950 7fdc8db37580 10 cache get: name=default.rgw.data.root++BUCKETNAME : type miss (requested=0x11, cached=0x16) 2019-09-05 20:08:13.950 7fdc8db37580 10 cache put: name=default.rgw.data.root++BUCKETNAME info.flags=0x11 2019-09-05 20:08:13.950 7fdc8db37580 10 moving default.rgw.data.root++BUCKETNAME to cache LRU end 2019-09-05 20:08:13.950 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : miss 2019-09-05 20:08:13.951 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x16 2019-09-05 20:08:13.951 7fdc8db37580 10 adding default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.951 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : type miss (requested=0x13, cached=0x16) 2019-09-05 20:08:13.951 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x13 2019-09-05 20:08:13.951 7fdc8db37580 10 moving default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry: cache_locator=default.rgw.data.root++BUCKETNAME 2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry: cache_locator=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4 fe0-bbe4-fdc16f590a82.16313306.1 2019-09-05 20:08:13.951 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : type miss (requested=0x16, cached=0x13) 2019-09-05 20:08:13.952 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x16 2019-09-05 20:08:13.952 7fdc8db37580 10 moving default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.952 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : hit (requested=0x11, cached=0x17) 2019-09-05 20:08:13.952 7fdc8db37580 10 cls_bucket_list_ordered BUCKETNAME[48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3856921.7] start _multipart_[] num_entries 1001 2019-09-05 20:08:13.960 7fdc8db37580 2 removed watcher, disabling cache Any recomendations for forcé recreate bucket index? Regards Manuel

1 1

Heavily-linked lists.ceph.com pipermail archive now appears to lead to 404s
by Florian Haas 05 Sep '19

05 Sep '19

Hi, is there any chance the list admins could copy the pipermail archive from lists.ceph.com over to lists.ceph.io? It seems to contain an awful lot of messages referred elsewhere by their archive URL, many (all?) of which appear to now lead to 404s. Example: google "Set existing pools to use hdd device class only". The top hit is a link to http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html: $ curl -IL http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html HTTP/1.1 301 Moved Permanently Server: nginx/1.10.3 (Ubuntu) Date: Thu, 29 Aug 2019 12:48:13 GMT Content-Type: text/html Content-Length: 194 Connection: keep-alive Location: https://lists.ceph.io/pipermail/ceph-users-ceph.com/2018-August/029078.html Strict-Transport-Security: max-age=31536000 HTTP/1.1 404 Not Found Server: nginx Date: Thu, 29 Aug 2019 12:48:14 GMT Content-Type: text/html; charset=utf-8 Content-Length: 3774 Connection: keep-alive X-Frame-Options: SAMEORIGIN Vary: Accept-Language, Cookie Content-Language: en Or maybe this is just a redirect rule that needs to be cleverer or more specific, rather than the apparent catch-all .com/.io redirect? Cheers, Florian

5 7

disk failure
by solarflow99 05 Sep '19

05 Sep '19

One of the things i've come to notice is when HDD drives fail, they often recover in a short time and get added back to the cluster. This causes the data to rebalance back and forth, and if I set the noout flag I get a health warning. Is there a better way to avoid this?

4 6

Re: CephFS+NFS For VMWare
by Maged Mokhtar 05 Sep '19

05 Sep '19

this is an old thread, but could be useful for others, i found out the discrepancy in VMware vmotion speed under iSCSI is probably due the "emulate_3pc" config attribute for the LIO target. if set to 0, then yes VMWare will issue io in 64KB blocks, so the bandwidth will indeed be around 25 MB/s. If emulate_3pc is set to 1, this will trigger VMWare to use vaai extended copy, which activates LIO's xcopy functionality which uses 512KB block sizes by default. We also bumped the xcopy block size to 4M (rbd object size) which gives around 400 MB/s vmotion speed, the same speed can also be achieved via Veeam backups. /Maged On 02/07/2018 14:36, Maged Mokhtar wrote: > > Hi Nick, > > With iSCSI we reach over 150 MB/s vmotion for single vm, 1 GB/s for > 7-8 vm migrations. Since these are 64KB block sizes, latency/iops is a > large factor, you need either controllers with write back cache or all > flash . hdds without write cache will suffer even with external wal/db > on ssds, giving around 80 MB/s vmotion migration. Potentially it may > be possible to get higher vmotion speeds by using fancy striping but i > would not recommend this unless your total queue depths in all your > vms is small compared to the number of osds. > > Regarding thin provisioning, a vmdk provisioned as lazy zeroed does > have an "initial" large impact on random write performance, could be > up to 10x slower. If you are writing a random 64KB to an un-allocated > vmfs block, vmfs will first write 1MB to fill the block with zeros > then write the 64KB client data, so although a lot of data is being > written the perceived client bandwidth is very low. The performance > will gradually get better with time until the disk is fully > provisioned. It is also possible to thick eager zero the vmdk disk at > creation time. Again this is more apparent with random writes rather > than sequential or vmotion load. > > Maged > > On 2018-06-29 18:48, Nick Fisk wrote: > >> This is for us peeps using Ceph with VMWare. >> >> My current favoured solution for consuming Ceph in VMWare is via >> RBD’s formatted with XFS and exported via NFS to ESXi. This seems to >> perform better than iSCSI+VMFS which seems to not play nicely with >> Ceph’s PG contention issues particularly if working with thin >> provisioned VMDK’s. >> >> I’ve still been noticing some performance issues however, mainly >> noticeable when doing any form of storage migrations. This is largely >> due to the way vSphere transfers VM’s in 64KB IO’s at a QD of 32. >> vSphere does this so Arrays with QOS can balance the IO easier than >> if larger IO’s were submitted. However Ceph’s PG locking means that >> only one or two of these IO’s can happen at a time, seriously >> lowering throughput. Typically you won’t be able to push more than >> 20-25MB/s during a storage migration >> >> There is also another issue in that the IO needed for the XFS journal >> on the RBD, can cause contention and effectively also means every NFS >> write IO sends 2 down to Ceph. This can have an impact on latency as >> well. Due to possible PG contention caused by the XFS journal updates >> when multiple IO’s are in flight, you normally end up making more and >> more RBD’s to try and spread the load. This normally means you end up >> having to do storage migrations…..you can see where I’m getting at here. >> >> I’ve been thinking for a while that CephFS works around a lot of >> these limitations. >> >> 1.It supports fancy striping, so should mean there is less per object >> contention >> >> 2.There is no FS in the middle to maintain a journal and other >> associated IO >> >> 3.A single large NFS mount should have none of the disadvantages seen >> with a single RBD >> >> 4.No need to migrate VM’s about because of #3 >> >> 5.No need to fstrim after deleting VM’s >> >> 6.Potential to do away with pacemaker and use LVS to do active/active >> NFS as ESXi does its own locking with files >> >> With this in mind I exported a CephFS mount via NFS and then mounted >> it to an ESXi host as a test. >> >> Initial results are looking very good. I’m seeing storage migrations >> to the NFS mount going at over 200MB/s, which equates to several >> thousand IO’s and seems to be writing at the intended QD32. >> >> I need to do more testing to make sure everything works as intended, >> but like I say, promising initial results. >> >> Further testing needs to be done to see what sort of MDS performance >> is required, I would imagine that since we are mainly dealing with >> large files, it might not be that critical. I also need to consider >> the stability of CephFS, RBD is relatively simple and is in use by a >> large proportion of the Ceph community. CephFS is a lot easier to >> “upset”. >> >> Nick >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users(a)lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >

1 0

Best osd scenario + ansible config?
by Yoann Moulin 05 Sep '19

05 Sep '19

Hello, I am deploying a new Nautilus cluster and I would like to know what would be the best OSD's scenario config in this case : 10x 6TB Disk OSDs (data) 2x 480G SSD previously used for journal and can be used for WAL and/or DB Is it better to put all WAL on one SSD and all DBs on the other one? Or put WAL and DB of the first 5 OSDs on the first SSD and the 5 others on the second one. A more general question, what is the impact on an OSD if we lose the WAL? The DB? Both? I plan to use EC 7+5 on 12 servers and I am OK if I lose one server temporarily. I have spare servers and I can easily add another one in this cluster. To deploy this cluster, I use ceph-ansible (stable-4.0). I am not sure how to configure the playbook to use SSD and disks with LVM. https://github.com/ceph/ceph-ansible/blob/master/docs/source/osds/scenarios… Is this good? osd_objectstore: bluestore lvm_volumes: - data: data-lv1 data_vg: data-vg1 db: db-lv1 db_vg: db-vg1 wal: wal-lv1 wal_vg: wal-vg1 - data: data-lv2 data_vg: data-vg2 db: db-lv2 db_vg: db-vg2 wal: wal-lv2 wal_vg: wal-vg2 Is it possible to let the playbook configure LVM for each disk in a mixed case? It looks like I must configure LVM before running the playbook but I am not sure if I missed something. Is wal_vg and db_vg can be identical (on VG per SSD shared with multiple OSDs)? Thanks for your help. Best regards, -- Yoann Moulin EPFL IC-IT

5 12

ceph fs crashes on simple fio test
by Frank Schilder 05 Sep '19

05 Sep '19

I need to harden our ceph cluster to satisfy the following properties: Assuming all hardware is functioning properly, 1) Cluster health has highest priority. Heartbeats have priority over client requests. 2) The cluster does not accept more IO than the OSDs can handle. The only exception might be a configurable burst option. 3) Client IO is accepted as long as it does not compromise 1. 4) Ideally, there is fair sharing of the cluster's IO budget between clients (like deadline or completely fair scheduling). Rogue clients should not get priority just because they push a lot. Unfortunately, with default settings our (any?) cluster prefers client IO over cluster health, which opens up for a simple but serious non-privileged client attack on cluster health: I observed a serious issue on our cluster when running a simple fio test that does 4K random writes on 100GB files (see details below). What I observe is that within a few seconds the cluster goes to health_warn with the MDS reporting slow meta data IO and behind on trimming. What is not shown in ceph health detail is, that all OSDs report thousands of slow ops and the counter increases really fast (I include some snippets below). This goes rapidly to the point where OSDs start missing heartbeats and start flapping, some PGs become inactive+degraded and start peering regularly. CPU load, memory and even network load were rather low (one-figure % CPU load). Nothing out of the ordinary. The mons had no issues. After the fio test completed, the OSDs slowly crunched through the backlog and managed to complete all OPS. The complete processing of the ops of a 30 second fio test took ca. 10 minutes. However, even then the cluster did not come back healthy as essential messages between daemons seem to have been lost. It is not a long stretch to assume that one can destroy a ceph fs beyond repair when running this test or an application performing the same IO pattern from multiple clients for several hours. We have a 500 node cluster as clients and I'm afraid that even ordinary IO might trigger this scenario in unlucky circumstances. Since our cluster is only half-trusted (we have root access to clients, but no control over user IO patterns), we are in need to harden our cluster against such destructive IO patterns as much as possible. To me it looks like the cluster is accepting way more IO than the OSDs can handle. Ideally, what I would like to do is configure effective rate limiting on (rogue) clients depending on how much OPS they have in flight already. I would expect that there are tunables for MDS/OSD daemons that control how much IO requests a client can submit/OSD will accept before throttling IO. In particular, I would like to prioritize heartbeats to prevent load-induced OSD flapping. How can I tune the cluster to satisfy the conditions outlined at the top of this e-mail? There were recent threads with similar topics, in particular, "MDS failing under load with large cache sizes" and others reporting unstable MDS daemons under load. However, I believe they were mostly related to cache trimming issues due to large amounts of files created. This is not the case here, its just 4 files with lots of random IO from a single client. A bit of information about our cluster and observations: The cluster is bluestore-only with 8+2 EC fs data pool on spinning disks and an 3(2) replicated fs meta data pool on SSD. We have 8 OSD hosts with 2 shards per host. Each host has 4SSDs and 12 10TB HDDs SAS 12GB with 4k block size. Network is 2x10G bonded for client and 2x10G bonded for replication. The replication network will be extended to 4x10G soon. We are aware that currently the network bandwidth greatly exceeds what the spinning disks can handle. It is dimensioned for adding more disks in the future. The fio job script is: [global] name=fio-rand-write directory=/home/fio # /home is on ceph fs filename_format=tmp/fio-$jobname-${HOSTNAME}-$jobnum-$filenum rw=randwrite bs=4K numjobs=4 time_based=1 runtime=30 [file1] size=100G ioengine=sync Its one of the examples in the fio source repo with small modifications (file pattern, run time). Shortly after starting this fio job from a client connected with single 10G line, all OSDs start reporting slow ops. Picking one, the log messages look like this: Aug 22 12:06:22 ceph-09 ceph-osd: 2019-08-22 10:06:22.151 7f399d1bd700 -1 osd.125 3778 get_health_metrics reporting 146 slow ops, oldest is osd_op(client.1178165.0:1670940 5.3fs0 5:fcc65fe3:::10000f403bc.0000063e:head [write 307200~4096,write 315392~8192,write 380928~12288,write 405504~12288,write 421888~4096,write 458752~4096,write 466944~4096,write 475136~4096,write 487424~8192,write 512000~4096,write 524288~4096,write 565248~4096,write 589824~8192,write 622592~4096,write 651264~4096,write 724992~12288] snapc 12e=[] ondisk+write+known_if_redirected e3778) ... Aug 22 12:06:49 ceph-09 journal: 2019-08-22 10:06:49.415 7f399d1bd700 -1 osd.125 3779 get_health_metrics reporting 4595 slow ops, olde st is osd_op(client.1178165.0:1686216 5.acs0 5:354e96d5:::10000f403bc.00000bc1:head [write 2359296~4096,write 2375680~4096,write 23838 72~4096,write 2404352~4096,write 2428928~8192,write 2469888~4096,write 2490368~8192,write 2514944~4096,write 2527232~4096,write 253542 4~4096,write 2588672~4096,write 2600960~4096,write 2621440~8192,write 2658304~4096,write 2715648~8192,write 2727936~4096] snapc 12e=[] ondisk+write+known_if_redirected e3778) ... Aug 22 12:12:57 ceph-09 journal: 2019-08-22 10:12:57.650 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 8419 slow ops, oldest is osd_op(client.1178165.0:2009417 5.3fs0 5:fcdcf2bd:::10000f47e2d.00001501:head [write 2236416~4096,write 2256896~4096,write 2265088~4096,write 2301952~4096,write 2322432~4096,write 2355200~4096,write 2371584~4096,write 2387968~4096,write 2449408~4096,write 2486272~4096,write 2547712~8192,write 2617344~4096,write 2809856~4096,write 3018752~4096,write 3194880~4096,write 3223552~4096] snapc 12e=[] ondisk+write+known_if_redirected e3839) Aug 22 12:12:58 ceph-09 ceph-osd: 2019-08-22 10:12:58.681 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 8862 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+write+known_if_redirected+full_force e3839) ... Aug 22 12:13:27 ceph-09 journal: 2019-08-22 10:13:27.691 7f399d1bd700 -1 osd.125 3839 get_health_metrics reporting 13795 slow ops, oldest is osd_op(mds.0.16909:69577573 5.e6s0 5.d8994de6 (undecoded) ondisk+write+known_if_redirected+full_force e3839) ... Aug 22 12:13:59 ceph-09 ceph-osd: 2019-08-22 10:13:59.762 7f399d1bd700 -1 osd.125 3900 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+retry+write+known_if_redirected+full_force e3875) ... Aug 22 12:14:46 ceph-09 journal: 2019-08-22 10:14:46.569 7f399d1bd700 -1 osd.125 3916 get_health_metrics reporting 969 slow ops, oldes t is osd_op(mds.0.16909:69577511 5.ees0 5.992388ee (undecoded) ondisk+retry+write+known_if_redirected+full_force e3875) Aug 22 12:14:47 ceph-09 ceph-osd: 2019-08-22 10:14:47.617 7f399d1bd700 -1 osd.125 3935 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5:7711c499:::10000f4798b.00000000:head [create,setxattr parent (289),setxattr layout (30)] snap c 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+full_force e3875) ... Aug 22 12:14:53 ceph-09 journal: 2019-08-22 10:14:53.675 7f399d1bd700 -1 osd.125 3939 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.16909:69577511 5.ees0 5:7711c499:::10000f4798b.00000000:head [create,setxattr parent (289),setxattr layout (30)] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+full_force e3875) This is the last log message, the OSD seems to have executed all OPS at this point or shortly after. The cluster state went to health_ok - at least shortly. At about 12:45 (yes, lunch break) we looked at the cluster again and it was back in health_warn with the following status output: [root@ceph-01 ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN 1 MDSs report slow metadata IOs 2 MDSs behind on trimming services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-02, ceph-03 mds: con-fs-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 192 osds: 192 up, 192 in data: pools: 7 pools, 790 pgs objects: 9.01 M objects, 16 TiB usage: 20 TiB used, 1.3 PiB / 1.3 PiB avail pgs: 790 active+clean io: client: 1.9 MiB/s rd, 21 MiB/s wr, 60 op/s rd, 721 op/s wr [root@ceph-01 ~]# ceph health detail HEALTH_WARN 1 MDSs report slow metadata IOs; 2 MDSs behind on trimming MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs mdsceph-08(mds.0): 2 slow metadata IOs are blocked > 30 secs, oldest blocked for 3275 secs MDS_TRIM 2 MDSs behind on trimming mdsceph-08(mds.0): Behind on trimming (1778/128) max_segments: 128, num_segments: 1778 mdsceph-12(mds.0): Behind on trimming (1780/128) max_segments: 128, num_segments: 1780 Num_segments was increasing constantly. Apparently, an operation got stuck and never completed. We hunted a little bit and found [root@ceph-mds:ceph-08 /]# ceph daemon mds.ceph-08 objecter_requests { "ops": [ { "tid": 71206302, "pg": "4.952611b7", "osd": 19, "object_id": "200.0001be74", "object_locator": "@4", "target_object_id": "200.0001be74", "target_object_locator": "@4", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 11:12:41.0.565912s", "osd_ops": [ "write 890478~1923" ] }, { "tid": 71206303, "pg": "4.952611b7", "osd": 19, "object_id": "200.0001be74", "object_locator": "@4", "target_object_id": "200.0001be74", "target_object_locator": "@4", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 11:12:41.0.566236s", "osd_ops": [ "write 892401~1931" ] }, { "tid": 71206301, "pg": "5.eeb918c6", "osd": 67, "object_id": "10000f26f67.00000000", "object_locator": "@5", "target_object_id": "10000f26f67.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "12e=[]", "mtime": "1970-01-01 00:00:00.000000s", "osd_ops": [ "trimtrunc 81854@573" ] }, { "tid": 69577573, "pg": "5.d8994de6", "osd": 125, "object_id": "10000f479c9.00000000", "object_locator": "@5", "target_object_id": "10000f479c9.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.256058s", "osd_ops": [ "create", "setxattr parent (319)", "setxattr layout (30)" ] }, { "tid": 69577598, "pg": "5.deb003e6", "osd": 125, "object_id": "10000f479da.00000000", "object_locator": "@5", "target_object_id": "10000f479da.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.258824s", "osd_ops": [ "create", "setxattr parent (288)", "setxattr layout (30)" ] }, { "tid": 71206300, "pg": "5.5cd5b20b", "osd": 163, "object_id": "10000f01396.00000000", "object_locator": "@5", "target_object_id": "10000f01396.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "268516s", "attempts": 1, "snapid": "head", "snap_context": "12e=[]", "mtime": "1970-01-01 00:00:00.000000s", "osd_ops": [ "trimtrunc 208782@573" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } Notice the ops from the 1970's. Checking ops and dump_blocked_ops on osd.19 showed that these lists were empty. So, we decided to restart osd.19 and it cleared out most of the stuck requests, but did not clear the health warnings: [root@ceph-mds:ceph-08 /]# ceph daemon mds.ceph-08 objecter_requests { "ops": [ { "tid": 69577573, "pg": "5.d8994de6", "osd": 125, "object_id": "10000f479c9.00000000", "object_locator": "@5", "target_object_id": "10000f479c9.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.256058s", "osd_ops": [ "create", "setxattr parent (319)", "setxattr layout (30)" ] }, { "tid": 69577598, "pg": "5.deb003e6", "osd": 125, "object_id": "10000f479da.00000000", "object_locator": "@5", "target_object_id": "10000f479da.00000000", "target_object_locator": "@5", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "265042s", "attempts": 5, "snapid": "head", "snap_context": "0=[]", "mtime": "2019-08-22 10:12:05.0.258824s", "osd_ops": [ "create", "setxattr parent (288)", "setxattr layout (30)" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } Restarting osd.125 finally resolved the health issues. However, the client I run fio on had lost connection to ceph due to this incident, which is really annoying. This client is the head node of our HPC cluster and it was not possible to restore ceph fs access without reboot. This is an additional bad fallout as all users will loose access to our HPC cluster when this happens (/home is on the ceph fs). I dumped the dump_historic_slow_ops of osd.125 in case anyone can use this information. I might be able to repeat this experiment, but cannot promise anything. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 14

bluestore_default_buffered_write
by Fyodor Ustinov 05 Sep '19

05 Sep '19

Hi! Сan anybody help me - if I turn on bluestore_default_buffered_write will i get a WriteBack or WriteThrow? According to the documentation, we don’t understand this. And the second question - but in general there is an analog of the writeback in the OSD (I perfectly understand the danger of such a cache). WBR, Fyodor.

1 0

CEPH 14.2.3
by Fyodor Ustinov 05 Sep '19

05 Sep '19

Dear CEPH Developers! We all respect you for your work. But I have one small request. Please, make an announcement about the new version and prepare the documentation before posting the new version to the repository. It is very, very, very necessary. WBR, Fyodor.

5 8

rados + radosstriper puts fail with "large" input objects (mimic/nautilus, ec pool)
by aoanla＠gmail.com 05 Sep '19

05 Sep '19

Hi everyone: I asked about this on the #ceph IRC channel, but didn't get much traction (and, as an aside: the advertised host for the channel's logs turns up unaccessible to me...). I have a new Ceph cluster presenting an erasure-coded pool. Current configuration is 8 nodes, each hosting a mon, and 1 osd per hdd (20 10TB HGST spinning disks per node) for a total of 160 OSDs in the cluster. The cluster configures fine with ceph-ansible (as nautilus or mimic), and ceph health is always marked as good (other than the "warn" for not having associated an application string with the current test pool). rados bench maxes out the bandwidth of the network interface when I try it with 4MB objects. However, attempting a more "real-world" test of rados -p ecpool --striper put obj 120MBfile causes the transfer to fail with "operation not permitted (95)" Inspection reveals that 3 stripe chunks get created - the first being the expected size, and the second and third being only a few kb in size. Object metadata from rados -p --striper ls obj is inconsistent with the sum of the on-disk size of the chunks. Can you advise how to diagnose what's breaking here? Thanks Sam Skipsey University of Glasgow

3 6

Proposal to disable "Avoid Duplicates" on all ceph.io lists
by Ilya Dryomov 05 Sep '19

05 Sep '19

Hello, Yesterday I copied dgallowa in "Heavily-linked lists.ceph.com pipermail archive now appears to lead to 404s" on ceph-users, but one of the subscribers reached out to me saying that they do not see him on the CC. This looks like a feature of mailman that has bitten others before: https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg02639.html > > mailman has an (extremely annoying, in my opinion) habit of munging cc: > > lines to drop the name of any recipient who has set their list delivery > > options to avoid duplicate messages where the recipient is in cc. > > Ultimately, the end user still gets the messages (reply-to-all gets the > > message back to the list, even though it drops the cc), but at the > > expense of not directly going to the inbox as desired. At least I still > > only get one copy of the message, but yes, I'd rather have that one copy > > come directly to me for all messages in the thread, rather than through > > the list because I was munged out of cc. > > > > I hope that mailman3/hyperkitty will have saner defaults and not munge > > cc lists to exclude subscribers merely based on their preference on > > duplicate mail receipt. > > Wow! You are right: > > I checked my @redhat.com inbox and I see address@hidden has > Kevin in CC. > > I checked my @gmail.com inbox (subscribed to mailing list) and I see > address@hidden does not have Kevin in CC. > > I guess my @redhat.com email was received directly from Max because I > was in CC list. It didn't pass through Mailman. I saw the original, > unmodified list of CCs with Kevin included. > > The @gmail.com email was received from the mailing list. Mailman did > what you described. > > This means I cannot send "Please CC maintainer" emails anymore because > I cannot be sure whether the maintainer was CCed! mailman3 that all ceph.io lists were transitioned to is apparently slightly saner in that this functionality is implemented as a handler, which can (hopefully!) be disabled by removing it from the pipeline: https://gitlab.com/mailman/mailman/blob/master/src/mailman/handlers/avoid_d… https://gitlab.com/mailman/mailman/blob/master/src/mailman/handlers/docs/av… I'm proposing that we disable this handler for all ceph.io lists. It confuses everyone and it's not clear if it is actually doing any good to anyone. gmail and other similar systems already deduplicate without munging the headers for everyone and folks who are filtering locally probably already have a generic rule for discarding duplicates. IMO the mailing list software should never touch To or Cc. Thoughts? Thanks, Ilya

4 3

v14.2.3 Nautilus rpm dependency problem: ceph-selinux-14.2.3-0.el7.x86_64 Requires: selinux-policy-base >= 3.13.1-229.el7_6.15
by Ning Li 05 Sep '19

05 Sep '19

Hi list: When I yum install ceph-14.2.3, I run into the following depenency issue: --> Processing Dependency: selinux-policy-base >= 3.13.1-229.el7_6.15 for package: 2:ceph-selinux-14.2.3-0.el7.x86_64 ---> Package groff-base.x86_64 0:1.22.2-8.el7 will be installed ---> Package libnetfilter_conntrack.x86_64 0:1.0.6-1.el7_3 will be installed ---> Package libnfnetlink.x86_64 0:1.0.1-4.el7 will be installed ---> Package perl-HTTP-Tiny.noarch 0:0.033-3.el7 will be installed ---> Package perl-parent.noarch 1:0.225-244.el7 will be installed ---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed --> Finished Dependency Resolution Error: Package: 2:ceph-selinux-14.2.3-0.el7.x86_64 (ceph) Requires: selinux-policy-base >= 3.13.1-229.el7_6.15 Available: selinux-policy-minimum-3.13.1-229.el7.noarch (base) selinux-policy-base = 3.13.1-229.el7 Available: selinux-policy-mls-3.13.1-229.el7.noarch (base) selinux-policy-base = 3.13.1-229.el7 Available: selinux-policy-targeted-3.13.1-229.el7.noarch (base) selinux-policy-base = 3.13.1-229.el7 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles —nodigest I was using the following yum repo which contains the lates rpms. - centos: https://mirrors.aliyun.com/centos/7.6.1810/ - ceph: https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/ I can’t find selinux-policy-base in cenots 7.6 repo and just find those selinux-policy.noarch : SELinux policy configuration selinux-policy-devel.noarch : SELinux policy devel selinux-policy-doc.noarch : SELinux policy documentation selinux-policy-minimum.noarch : SELinux minimum base policy selinux-policy-mls.noarch : SELinux mls base policy selinux-policy-sandbox.noarch : SELinux policy sandbox selinux-policy-targeted.noarch : SELinux targeted base policy Is there anything broken about the new released ceph-14.2.3 or anything else I need to do?

1 1

ceph cluster warning after adding disk to cluster
by Amudhan P 04 Sep '19

04 Sep '19

Hi, I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning. My current setup: 3 OSD node each with a single disk, recently I added one more disk in one of the node and ceph cluster status showing warning. I can see the progress but it was more than 12 hours but still its moving objects. How to increase the speed of moving objects? output from "ceph -s" cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN 834820/7943361 objects misplaced (10.510%) services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 4 osds: 4 up, 4 in; 12 remapped pgs data: pools: 2 pools, 64 pgs objects: 2.65 M objects, 178 GiB usage: 548 GiB used, 6.7 TiB / 7.3 TiB avail pgs: 834820/7943361 objects misplaced (10.510%) 52 active+clean 11 active+remapped+backfill_wait 1 active+remapped+backfilling io: recovery: 0 B/s, 6 objects/s output from "ceph osd df " ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 0 hdd 1.81940 1.00000 1.8 TiB 88 GiB 1.7 TiB 4.71 0.64 40 3 hdd 1.81940 1.00000 1.8 TiB 96 GiB 1.7 TiB 5.15 0.70 24 1 hdd 1.81940 1.00000 1.8 TiB 182 GiB 1.6 TiB 9.79 1.33 64 2 hdd 1.81940 1.00000 1.8 TiB 182 GiB 1.6 TiB 9.79 1.33 64 TOTAL 7.3 TiB 548 GiB 6.7 TiB 7.36 MIN/MAX VAR: 0.64/1.33 STDDEV: 2.43 regards Amudhan P

4 4

Slow peering caused by "wait for new map"
by Bryan Stillwell 04 Sep '19

04 Sep '19

Our test cluster is seeing a problem where peering is going incredibly slow shortly after upgrading it to Nautilus (14.2.2) from Luminous (12.2.12). From what I can tell it seems to be caused by "wait for new map" taking a long time. When looking at dump_historic_slow_ops on pretty much any OSD I see stuff like this: # ceph daemon osd.112 dump_historic_slow_ops [...snip...] { "description": "osd_pg_create(e180614 287.4b:177739 287.75:177739 287.1c3:177739 287.1cf:177739 287.1e1:177739 287.2dd:177739 287.2fc:177739 287.342:177739 287.382:177739)", "initiated_at": "2019-09-03 15:12:41.366514", "age": 4800.8847047119998, "duration": 4780.0579745630002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-09-03 15:12:41.366514", "event": "initiated" }, { "time": "2019-09-03 15:12:41.366514", "event": "header_read" }, { "time": "2019-09-03 15:12:41.366501", "event": "throttled" }, { "time": "2019-09-03 15:12:41.366547", "event": "all_read" }, { "time": "2019-09-03 15:39:03.379456", "event": "dispatched" }, { "time": "2019-09-03 15:39:03.379477", "event": "wait for new map" }, { "time": "2019-09-03 15:39:03.522376", "event": "wait for new map" }, { "time": "2019-09-03 15:53:55.912499", "event": "wait for new map" }, { "time": "2019-09-03 15:59:37.909063", "event": "wait for new map" }, { "time": "2019-09-03 16:00:43.356023", "event": "wait for new map" }, { "time": "2019-09-03 16:20:50.575498", "event": "wait for new map" }, { "time": "2019-09-03 16:31:48.689415", "event": "started" }, { "time": "2019-09-03 16:32:21.424489", "event": "done" } ] } It always seems to be in osd_pg_create() with multiple "wait for new map" messages before it finally does something. What could be causing it so long to get the OSD map? The mons don't appear to be overloaded in any way. Thanks, Bryan

2 5

v14.2.3 Nautilus released
by Abhishek Lekshmanan 04 Sep '19

04 Sep '19

This is the third bug fix release of Ceph Nautilus release series. This release fixes a security issue. We recommend all Nautilus users upgrade to this release. For upgrading from older releases of ceph, general guidelines for upgrade to nautilus must be followed Notable Changes --------------- * CVE-2019-10222 - Fixed a denial of service vulnerability where an unauthenticated client of Ceph Object Gateway could trigger a crash from an uncaught exception * Nautilus-based librbd clients can now open images on Jewel clusters. * The RGW `num_rados_handles` has been removed. If you were using a value of `num_rados_handles` greater than 1, multiply your current `objecter_inflight_ops` and `objecter_inflight_op_bytes` parameters by the old `num_rados_handles` to get the same throttle behavior. * The secure mode of Messenger v2 protocol is no longer experimental with this release. This mode is now the preferred mode of connection for monitors. * "osd_deep_scrub_large_omap_object_key_threshold" has been lowered to detect an object with large number of omap keys more easily. For a detailed changelog please refer to the official release notes entry at the ceph blog: https://ceph.io/releases/v14-2-3-nautilus-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.3.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 0f776cf838a1ae3130b2b73dc26be9c95c6ccc39 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

3 2

Bucket policies with OpenStack integration and limiting access
by shubjero 04 Sep '19

04 Sep '19

Good day, We have a Ceph cluster and make use of object-storage and integrate with OpenStack. Each OpenStack project/tenant is given a radosgw user which allows all keystone users of that project to access the object-storage as that single radosgw user. The radosgw user is the project id of the OpenStack project/tenant. Sometimes we have use cases where we want to access the object-storage outside of the swift-api and use tools like the aws-cli or homebrew java applications to access the object storage. For this use case what we do is generate S3 access/secret key for the specific radosgw user and they have full access to the object storage for that OpenStack project/tenant. What we want to know is if it is possible to provide granular access to containers within a single OpenStack project using S3 access keys or S3 sub-users? I know that the Swift API has ACL's that can limit by keystone user but we are exploring the possibility of doing this using S3 and S3 bucket policies so that the tools our team are developing (open source) are more transferrable to AWS S3 and Rados GW. Thanks all, Jared Baker Cloud Architect, OICR

1 0

Strange hardware behavior
by Fyodor Ustinov 04 Sep '19

04 Sep '19

Hi! I understand that this question is not quite for this mailing list, but nonetheless, experts who may be encountered this have gathered here. I have 24 servers, and on each, after six months of work, the following began to happen: [root@S-26-5-1-2 cph]# uname -a Linux S-26-5-1-2 5.2.11-1.el7.elrepo.x86_64 #1 SMP Thu Aug 29 08:10:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.76334 s, 279 MB/s [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 4.54834 s, 231 MB/s sdc - SSD disk. sdd - HDD. It can be seen that ssd works somehow slowly, and hdd - too quickly. Reboot - nothing changes. And only poweroff/poweron cycle change behavior to normal: [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdc bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 3.24042 s, 324 MB/s [root@S-26-5-1-2 cph]# dd if=/dev/zero of=/dev/sdd bs=1M count=1000 oflag=sync 1048576000 bytes (1.0 GB) copied, 13.7709 s, 76.1 MB/s Absoluteli nothing in system and ceph log (this servers used for OSD) about that. Perhaps someone has encountered similar behavior? WBR, Fyodor.

8 12

Nautilus packaging on stretch
by mjclark.00＠gmail.com 04 Sep '19

04 Sep '19

Hello, I'm trying to install nautilus on stretch following the directions here https://docs.ceph.com/docs/master/install/get-packages/ . However, it seems the stretch repo only includes ceph-deploy. Are the rest of the packages missing on purpose or have I missed something obvious? Thanks

3 2

slow requests with the ceph osd dead lock?
by linghucongsong 04 Sep '19

04 Sep '19

Hi all! My ceph version is 10.2.11 and I use rgw and EC(7+3). When I use muliti clients to read and write on one rgw bucket .The bucket have 200 shards. There look like dead look on osd? Thanks for all! The osd block ops all have the similiar below logs and the block osds not always on the same osd. "description": "osd_op(client.3147989.0:2782587465 13.36f21cde .dir.517af746-28f1-454c-ba41-0c4fd51af270.896917.11.99 [call rgw.bucket_complete_op] snapc 0=[] ack+ondisk+write+known_if_redirected e26026)", "initiated_at": "2019-09-04 13:38:48.089766", "age": 3520.459786, "duration": 3521.432087, "type_data": [ "delayed", { "client": "client.3147989", "tid": 2782587465 }, [ { "time": "2019-09-04 13:38:48.089766", "event": "initiated" }, { "time": "2019-09-04 13:38:48.089794", "event": "queued_for_pg" }, { "time": "2019-09-04 13:38:48.180941", "event": "reached_pg" }, { "time": "2019-09-04 13:38:48.180962", "event": "waiting for rw locks" }, { "time": "2019-09-04 13:38:48.430598", "event": "reached_pg" }, { "time": "2019-09-04 13:38:48.430616", "event": "waiting for rw locks" }, { "time": "2019-09-04 13:38:49.150673", "event": "reached_pg" }, { "time": "2019-09-04 13:38:49.150691", "event": "waiting for rw locks" }, { "time": "2019-09-04 13:38:51.421887", "event": "reached_pg" }, { "time": "2019-09-04 13:38:51.421904", "event": "waiting for rw locks" }, { "time": "2019-09-04 13:38:51.990943", "event": "reached_pg" }, { "time": "2019-09-04 13:38:51.990961", "event": "waiting for rw locks" }, { "time": "2019-09-04 13:38:54.343921", "event": "reached_pg" }, { "time": "2019-09-04 13:38:54.343938", "event": "waiting for rw locks" }

2 1

rgw auth error with self region name
by 黄明友 04 Sep '19

04 Sep '19

hi,all: I use the aws s3 java sdk , when make a new bucket , with the hostname " s3.my-self.mydomain.com" ; will get a auth error. but , when I use the hostname " s3.us-east-1.mydomian.com" ,will be ok, why ? 黄明友 IT基础架构部经理 V.Photos 云摄影移动电话: +86 13540630430 客服电话：400 - 806 - 5775 电子邮件: hmy(a)v.photos 官方网址: www.v.photos 上海黄浦区中山东二路88号外滩SOHO3Q F栋 2层北京朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层广州天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区深圳南山区蛇口网谷科技大厦二期A座102网谷双创街 1层成都成华区建设路世贸广场 7层

2 1

ceph mons stuck in electing state
by Nick 04 Sep '19

04 Sep '19

Hello, I have an old ceph 0.94.10 cluster that had 10 storage nodes with one extra management node used for running commands on the cluster. Over time we'd had some hardware failures on some of the storage nodes, so we're down to 6, with ceph-mon running on the management server and 4 of the storage nodes. We attempted deploying a ceph.conf change and restarted ceph-mon and ceph-osd services, but the cluster went down on us. We found all the ceph-mons are stuck in the electing state, I can't get any response from any ceph commands but I found I can contact the daemon directly and get this information (hostnames removed for privacy reasons): root@<mgmt1>:~# ceph daemon mon.<mgmt1> mon_status { "name": "<mgmt1>", "rank": 0, "state": "electing", "election_epoch": 4327, "quorum": [], "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 10, "fsid": "69611c75-200f-4861-8709-8a0adc64a1c9", "modified": "2019-08-23 08:20:57.620147", "created": "0.000000", "mons": [ { "rank": 0, "name": "<mgmt1>", "addr": "[fdc4:8570:e14c:132d::15]:6789\/0" }, { "rank": 1, "name": "<mon1>", "addr": "[fdc4:8570:e14c:132d::16]:6789\/0" }, { "rank": 2, "name": "<mon2>", "addr": "[fdc4:8570:e14c:132d::28]:6789\/0" }, { "rank": 3, "name": "<mon3>", "addr": "[fdc4:8570:e14c:132d::29]:6789\/0" }, { "rank": 4, "name": "<mon4>", "addr": "[fdc4:8570:e14c:132d::151]:6789\/0" } ] } } Is there any way to force the cluster back into a quorum even if it's just one mon running to start it up? I've tried exporting the mgmt's monmap and injecting it into the other nodes, but it didn't make any difference. Thanks!

4 5

Fwd: Applications slow in VMs running RBD disks
by Gesiel Galvão Bernardes 04 Sep '19

04 Sep '19

Hi, Em qui, 29 de ago de 2019 às 22:32, fengyd <fengyd81(a)gmail.com> escreveu: > Hi, > > The issue is still there? > Yes, yet. > I have met an IO peformance issue recently and found that the count of the > max fd for the Qemu/KVM was not bigger enough, the fd for Qemu/KVM was > exhausted, the issue was solved after increasing the count of the max fd. > > How check and increase max fd for qemu? Can you give-me way? Regards, Gesiel > > On Wed, 21 Aug 2019 at 20:53, Gesiel Galvão Bernardes < > gesiel.bernardes(a)gmail.com> wrote: > >> Hi Eliza, >> >> Em qua, 21 de ago de 2019 às 09:30, Eliza <eli(a)chinabuckets.com> >> escreveu: >> >>> Hi >>> >>> on 2019/8/21 20:25, Gesiel Galvão Bernardes wrote: >>> > I`m use a Qemu/kvm(Opennebula) with Ceph/RBD for running VMs, and I >>> > having problems with slowness in aplications that many times not >>> > consuming very CPU or RAM. This problem affect mostly Windows. >>> Appearly >>> > the problem is that normally the application load many short files >>> (ex: >>> > DLLs) and these files take a long time to load, generating a slowness. >>> >>> Did you check/test your network connection? >>> Do you have a fast network setup? >> >> >> I have a bond of two 10GB interfaces, with little use. >> >>> >>> >> regards. >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users(a)lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> _______________________________________________ >> ceph-users mailing list >> ceph-users(a)lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >

2 1

Nautilus 14.2.3 packages appearing on the mirrors
by Sasha Litvak 04 Sep '19

04 Sep '19

Is there an actual release or an accident?

1 0

Re: forcing an osd down
by Frank Schilder 03 Sep '19

03 Sep '19

"ceph osd down" will mark an OSD down once, but not shut it down. Hence, it will continue to send heartbeats and request to be marked up again after a couple of seconds. To keep it down, there are 2 ways: - either set "ceph osd set noup", - or actually shut the OSD down. The first version will allow the OSD to keep running so you can talk to the daemon while it is marked "down" . Be aware that the OSD will be marked "out" after a while. You might need to mark it "in" manually when you are done with maintenance. I believe with nautilus it is possible to set the noup flag on a specific OSD, which is much safer. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: ceph-users <ceph-users-bounces(a)lists.ceph.com> on behalf of solarflow99 <solarflow99(a)gmail.com> Sent: 03 September 2019 19:40:59 To: Ceph Users Subject: [ceph-users] forcing an osd down I noticed this has happened before, this time I can't get it to stay down at all, it just keeps coming back up: # ceph osd down osd.48 marked down osd.48. # ceph osd tree |grep osd.48 48 3.64000 osd.48 down 0 1.00000 # ceph osd tree |grep osd.48 48 3.64000 osd.48 up 0 1.00000 health HEALTH_WARN 2 pgs backfilling 1 pgs degraded 2 pgs stuck unclean recovery 18/164089686 objects degraded (0.000%) recovery 1467405/164089686 objects misplaced (0.894%) monmap e1: 3 mons at {0=192.168.4.10:6789/0,1=192.168.4.11:6789/0,2=192.168.4.12:6789/0<http://192.168.4.10:6789/0,1=192.168.4.11:6789/0,2=192.168.4.12:6789/0>} election epoch 210, quorum 0,1,2 0,1,2 mdsmap e166: 1/1/1 up {0=0=up:active}, 2 up:standby osdmap e25733: 45 osds: 45 up, 44 in; 2 remapped pgs

2 1

MDS blocked ops; kernel: Workqueue: ceph-pg-invalid ceph_invalidate_work [ceph]
by Frank Schilder 03 Sep '19

03 Sep '19

Hi, I encountered a problem with blocked MDS operations and a client becoming unresponsive. I dumped the MDS cache, ops, blocked ops and some further log information here: https://files.dtu.dk/u/peQSOY1kEja35BI5/2010-09-03-mds-blocked-ops?l A user of our HPC system was running a job that creates a somewhat stressful MDS load. This workload tends to lead to MDS warnings like "slow metadata ops" and "client does not respond to caps release", which usually disappear without intervantion after a while. He cancelled the job and one operation from one of the clients remained stuck in the MDS. We had a health warning about 1 blocked meta data operation and one client failing to respond to caps release. I should mention that we execute "echo 3 > /proc/sys/vm/drop_caches" in the epilogue script executed after every job, which usually cleans up all unused caps without problems. So, at the time I was looking at the number of client caps, these were down to below 100 for the client in question due to epilogue script execution. Looks like there might be a race condition with the drop caches and MDS requests. In addition, while this happened, there was backfill going on. All PGs were active+other stuff. All storage was r/w-accessible. On the client side, this was in the logs: Sep 3 09:15:57 sn110 kernel: INFO: task kworker/0:1:79782 blocked for more than 120 seconds. Sep 3 09:15:57 sn110 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 3 09:15:57 sn110 kernel: kworker/0:1 D ffff995cf4614100 0 79782 2 0x00000000 Sep 3 09:15:57 sn110 kernel: Workqueue: ceph-pg-invalid ceph_invalidate_work [ceph] Sep 3 09:15:57 sn110 kernel: Call Trace: [... see link above ...] I did not see slow ops on any of the OSDs. All other information in the link above. We had to reboot the client to resolve this problem. It seems like the MDS does not clean up blocked requests in certain situations when it ought to be possible. I hope the cache and ops dumps help pinpoint the reason. Best regards, Frank

2 1

TASK_UNINTERRUPTIBLE kernel client threads
by Toby Darling 03 Sep '19

03 Sep '19

Hi We have a couple of RHEL 7.6 (3.10.0-957.21.3.el7.x86_64) clients that have a number of uninterruptible threads and I'm wondering if we're looking at the issue fixed by https://www.spinics.net/lists/ceph-devel/msg45467.html (the fix hasn't made it into RHEL 7.7 3.10.0-1062). Stack traces of the hung threads are at http://p.ip.fi/9pQA There are a number of entries listed in /sys/kernel/debug/ceph/*/{osdc,mdsc} at http://p.ip.fi/VVzx Unfortunately, the issue isn't consistently reproducible Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070

2 1

Manual pg repair help
by Marc Roos 03 Sep '19

03 Sep '19

Is there no ceph wiki page with examples of manual repairs with the ceph-objectstore-tool (eg. where pg repair and pg scrub don’t work) I am having this issue for quite some time. 2019-09-02 14:17:34.175139 7f9b3f061700 -1 log_channel(cluster) log [ERR] : deep-scrub 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing And tried to resolve it according to this procedure[0] but now I am getting the message ceph-objectstore-tool --dry-run --type bluestore --data-path /var/lib/ceph/osd/ceph-29 --pgid 17.36 '{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","snapid":-2, "hash":1357874486,"pool":17,"namespace":"","max":0}' remove Snapshots are present, use removeall to delete everything I am not sure about this removeall, but I do not want to start deleting snapshots hoping it will amount to something. Besides if only maybe 4mb block is damaged, do you really need to purge snapshots of 40GB. I rather have a snapshot of 40GB missing 4MB than having no snapshot at all. [0] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg47218.html PS. Is there a record of who is having the longest unhealthy cluster state? Because I would not like it to be me ;)

1 0

Bug: ceph-objectstore-tool ceph version 12.2.12
by Marc Roos 03 Sep '19

03 Sep '19

Max 2x listed ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}]

1 0

Placement Groups - default.rgw.metadata pool.
by Jaroslaw Owsiewski 03 Sep '19

03 Sep '19

Hi, we have cluster which running Ceph Luminous 12.2.12. Rados Gateway Only (S3). Pool with data is placed on SAS HDDs (1430 pcs) and the rest of pools is placed on the SSD (72 pcs) disks - 72 hosts with OSD role (3 rows, 2 racks per row, and 12 hosts per rack). BlueStore of course. The question is: how many PG we need for default.rgw.metadata? Any ideas? Example statistics from this cluster: pool default.rgw.buckets.data id 15 189946/9416348469 objects misplaced (0.002%) recovery io 13.5MiB/s, 83objects/s client io 531MiB/s rd, 51.4MiB/s wr, 13.24kop/s rd, 4.84kop/s wr pool .rgw.root id 16 nothing is going on pool default.rgw.control id 17 nothing is going on pool default.rgw.meta id 18 client io 47.0MiB/s rd, 0B/s wr, 57.95kop/s rd, 450op/s wr pool default.rgw.log id 19 nothing is going on pool default.rgw.buckets.index id 20 client io 3.12MiB/s rd, 0B/s wr, 3.19kop/s rd, 1.92kop/s wr Regards, Jarek

1 0

Re: pg 17.36 is active+clean+inconsistent head expected clone 1 missing?
by Marc Roos 03 Sep '19

03 Sep '19

Hi Steve, I was just about to follow your steps[0] with the ceph-objectstore-tool, (I do not want to remove more snapshots) So I have this error pg 17.36 is active+clean+inconsistent, acting [7,29,12] 2019-09-02 14:17:34.175139 7f9b3f061700 -1 log_channel(cluster) log [ERR] : deep-scrub 17.36 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head : expected clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 missing I removed the snapshot with snapshot id 4, did a pg repair without any result. I am trying to understand this command of yours, ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-229/ --pgid 2.9a6 '{"oid":"rb.0.2479b45.238e1f29","snapid":-2,"hash":2320771494,"max":0,"p ool":2,"namespace":"","max":0}' I think you are getting this info from the --op list not? And grep for the "rbd_data.1f114174b0dc51.0000000000000974" occurance? I have these entries on osd.29 ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":63,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] ["17.36",{"oid":"rbd_data.1f114174b0dc51.0000000000000974","key":"","sna pid":-2,"hash":1357874486,"max":0,"pool":17,"namespace":"","max":0}] So I guess snapid's with -2 are bad? I have noticed actually quite a few -2 listings in these op list output, and do not understand why there are so many and the cluster is healthy except for this pg 17.36. [0] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg47212.html -----Original Message----- From: Steve Anthony [mailto:sma310@lehigh.edu] Sent: vrijdag 16 november 2018 17:44 To: ceph-users(a)lists.ceph.com Subject: Re: [ceph-users] pg 17.36 is active+clean+inconsistent head expected clone 1 missing? Looks similar to a problem I had after a several OSDs crashed while trimming snapshots. In my case, the primary OSD thought the snapshot was gone, but some of the replicas are still there, so scrubbing flags it. First I purged all snapshots and then ran ceph pg repair on the problematic placement groups. The first time I encountered this, that action was sufficient to repair the problem. The second time however, I ended up having to manually remove the snapshot objects. http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-June/027431.html Once I had done that, repair the placement group fixed the issue. -Steve On 11/16/2018 04:00 AM, Marc Roos wrote: > > > I am not sure that is going to work, because I have this error quite > some time, from before I added the 4th node. And on the 3 node cluster > it was: > > osdmap e18970 pg 17.36 (17.36) -> up [9,0,12] acting [9,0,12] > > If I understand correctly what you intent to do, moving the data around. > This was sort of accomplished by adding the 4th node. > > > > -----Original Message----- > From: Frank Yu [mailto:flyxiaoyu@gmail.com] > Sent: vrijdag 16 november 2018 3:51 > To: Marc Roos > Cc: ceph-users > Subject: Re: [ceph-users] pg 17.36 is active+clean+inconsistent head > expected clone 1 missing? > > try to restart osd.29, then use pg repair. If this doesn't work or it > appear again after a while, scan your HDD which used for osd.29, maybe > there is bad sector of your disks, just replace the disk with new one. > > > > On Thu, Nov 15, 2018 at 5:00 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > Forgot, these are bluestore osds > > > > -----Original Message----- > From: Marc Roos > Sent: donderdag 15 november 2018 9:59 > To: ceph-users > Subject: [ceph-users] pg 17.36 is active+clean+inconsistent head > expected clone 1 missing? > > > > I thought I will give it another try, asking again here since there > is > another thread current. I am having this error since a year or so. > > This I of course already tried: > ceph pg deep-scrub 17.36 > ceph pg repair 17.36 > > > [@c01 ~]# rados list-inconsistent-obj 17.36 > {"epoch":24363,"inconsistents":[]} > > > [@c01 ~]# ceph pg map 17.36 > osdmap e24380 pg 17.36 (17.36) -> up [29,12,6] acting [29,12,6] > > > [@c04 ceph]# zgrep ERR ceph-osd.29.log*gz > ceph-osd.29.log-20181114.gz:2018-11-13 14:19:55.766604 7f25a05b1700 > -1 > log_channel(cluster) log [ERR] : deep-scrub 17.36 > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:head > expected > clone 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4 1 > missing > ceph-osd.29.log-20181114.gz:2018-11-13 14:24:55.943454 7f25a05b1700 > -1 > log_channel(cluster) log [ERR] : 17.36 deep-scrub 1 errors > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- Steve Anthony LTS HPC Senior Analyst Lehigh University sma310(a)lehigh.edu _______________________________________________ ceph-users mailing list ceph-users(a)lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

BACKPORT #21481 - jewel: "FileStore.cc: 2930: FAILED assert(0 == "unexpected error")" in fs
by Reddi Prasad Yendluri 03 Sep '19

03 Sep '19

Hi, I have come across this issue on one of ceph cluster storage nodes. (in the production OpenStack). Now, I would like to redploy the CEPH and start OSD on this node again with the same version. I will try to upgrade to a newer version, after increasing the capacity of cluster storage and bring the ceph health status to "OK" status. Has any one used, kolla ansible to deploy or add new node the existing ceph cluster storage. let me know the steps. What I have tried: 1. Ran "ceph playbook from kolla-ansible $ kolla-ansible bootstrap-servers -i <path/multinode> --limit nodeXX.maas Output: success 2. Unable to go through the "prechecks" as docker deamon is not running. 3. Could not start the docker, Uninstalled docker and during installation of docker using the docker-install/site.yml the docker could not start docker neither install nor start the docker because it is looking for dependencies of OSD device (disk) 4. Trying to uninstall the docker completely from physical node and insgtall docker from scratch and reset up the CEPH on this node? Has anyone tried resetup CEPH OSD on the existing ceph cluster storge. Thanks, Reddi Prasad YENDLURI Cloud Specialist M +65 8345 9599 | D +65 6220 9908 Office: *51B Circular Road Singapore 049406* [image: PALO IT]

1 0

ceph-volume 'ascii' codec can't decode byte 0xe2
by changcheng.liu 03 Sep '19

03 Sep '19

Hi Alfredo Deza I see your below PR in ceph PR 23289 "ceph-volume ensure encoded bytes are always used" I meet with one problem when using ceph-deploy to deploy ceph cluster. During running ceph-volume command to create OSD through ceph-deploy, it always hit below problem: nstcc1@nstcloudcc1:deploy$ PYTHONIOENCODING=UTF8 ceph-deploy osd create --data /dev/sdc nstcloudcc1 [nstcloudcc1][WARNIN] stderr: purged osd.0 [nstcloudcc1][WARNIN] --> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 132: ordinal not in range(128) [nstcloudcc1][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdc [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs Do you know how to solve the problem? Below is the deploy host environment: nstcc1@nstcloudcc1:deploy$ cat /etc/os-release NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.1 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic nstcc1@nstcloudcc1:deploy$ env | grep 'LC_\|PYTHON' LC_ALL=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 PYTHONIOENCODING=UTF8 LC_NAME=en_US.UTF-8 LC_ADDRESS=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_TELEPHONE=en_US.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_TIME=en_US.UTF-8 nstcc1@nstcloudcc1:deploy$ python --version Python 2.7.15rc1 B.R. Changcheng

1 0

EC Compression
by Ashley Merrick 02 Sep '19

02 Sep '19

I have an EC RBD pool I want to add compression on. I have the meta pool and the data pool, do I need to enable compression on both for it to function correctly or only on one pool? Thanks

2 1

Which network is used for recovery / rebalancing
by Thomas Schneider 02 Sep '19

02 Sep '19

Hi, my ceph cluster is recovering / rebalancing PGs since some OSDs are marked out. Question: Which network is used for this task be default? Is there any configuration for the network to be used in //etc/ceph/ceph.conf/? THX

2 1

Fwd: Kinetic support
by Johan Thomsen 02 Sep '19

02 Sep '19

Hey, What is the current status of Kinetic KV support in Ceph? I'm asking because: https://www.crn.com.au/news/seagate-quietly-bins-open-storage-project-519345 .. and the fact that kinetic-cpp-client hasn't been updated in four years and only compiles against OpenSSL 1.0.2, which will become EOL by the end of 2019. Or am I totally wrong? ^ Thank you for reply in advance, /Johan

1 0

removing/flattening a bucket without data movement?
by Zoltan Arnold Nagy 02 Sep '19

02 Sep '19

Hi folks, Originally our osd tree looked like this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2073.15186 root default -14 176.63100 rack s01-rack -19 176.63100 host s01 <snip osds> -15 171.29900 rack s02-rack -20 171.29900 host s02 <snip osds> etc. You get the idea. It was a legacy thing as we've been upgrading this cluster since probably firefly, and started with way less hardware. The crush rule was set up like this originally: step take default step chooseleaf firstn 0 type rack which we have modified to step take default step chooseleaf firstn 0 type host taking advantage of chooseleaf's behavior (eg searching in depth instead of just a single level). Now we thought we could get rid of the rack buckets simply by moving the host buckets to the root using "ceph osd crush move s01 root=default", however this resulted in a bunch of data movement. Swapping the IDs manually in the crushmap seems to work (verified via crushtool's --compare), eg. changing the ID of s01 to s01-rack's and vice versa, including all shadow trees. Looking around I saw that there is a swap-bucket command but that does not swap the IDs just bucket contents, so would result in data movement. Other than manually editing the crushmap is there a better way to achieve this? Is this way the most optimal? Cheers, Zoltan

3 7

trouble with grafana dashboards in nautilus
by Rory Schramm 01 Sep '19

01 Sep '19

HI, I recently upgraded my cluster from 12.2 to 14.2 and I'm having some trouble getting the mgr dashboards for grafana working. I setup Prometheus and Grafana per https://docs.ceph.com/docs/nautilus/mgr/prometheus/#mgr-prometheus However, for the osd disk performance statistics graphs on the host details dashboard I'm getting the following error: "found duplicate series for the match group {device="dm-5", instance=":9100"} on the right hand-side of the operation: [{name="ceph_disk_occupation", ceph_daemon="osd.13", db_device="/dev/dm-8", device="dm-5", instance=":9100", job="ceph"}, {name="ceph_disk_occupation", ceph_daemon="osd.15", db_device="/dev/dm-10", device="dm-5", instance=":9100", job="ceph"}];many-to-many matching not allowed: matching labels must be unique on one side" This also happens on the following graphs: Host Overview/AVG Disk Utilization Host Details/OSD Disk Performance Statistics/* Also the following graphs show no data points: OSD Details/Physical Device Performance/* prometheus version: 2.12.0 node exporter: 0.15.2 grafana version: 6.3.3 note that my osds all have separate data and rocks db devices. I have also upgraded all the osds to nautilus via ceph-bluestore-tool repair. Any idea what's needed to fix this? Thanks below are the Prometheus config files prometheus.yml global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: - job_name: 'node' file_sd_configs: - files: - node_targets.yml - job_name: 'ceph' honor_labels: true file_sd_configs: - files: - ceph_targets.yml ---- node_targets.yml: [ { "targets": [ "nas-osd-01:9100" ], "labels": { "instance": "nas-osd-01" } }, { "targets": [ "nas-osd-02:9100" ], "labels": { "instance": "nas-osd-02" } }, { "targets": [ "nas-osd-02:9100" ], "labels": { "instance": "nas-osd-03" } } ] --- ceph_targets.yml: [ { "targets": [ "nas-osd-01:9283" ], "labels": {} }, { "targets": [ "nas-osd-02:9283" ], "labels": {} }, { "targets": [ "nas-osd-03:9283" ], "labels": {} } ]

1 0