- ceph-users - lists.ceph.io

log_channel(cluster) log [ERR] : Error -2 reading object
by Frank Schilder 05 Jun '20

05 Jun '20

Hi all, I found these messages today: 2020-06-04 17:07:57.471 7fa0aa16e700 -1 log_channel(cluster) log [ERR] : Error -2 reading object 14:e4c5ebb6:::1000203c59b.00000002:head 2020-06-04 17:08:04.236 7fa0aa16e700 -1 log_channel(cluster) log [ERR] : Error -2 reading object 14:e4c9a1a1:::1000203ad7f.00000000:head in one of our OSD logs. The disk is healthy according to smartctl. Should I worry about that? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

bad balacing (octopus)
by Ml Ml 05 Jun '20

05 Jun '20

Hello, any idea why it´s so bad balanced? e.g.: osd.52 (82%) vs osd.34 (29%) I did run "/usr/bin/ceph osd reweight-by-utilization " by cron for some time, since i was low on space for some time and that helped a bit. What should i do next? Here is some info: root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_OK services: mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 4h) mgr: ceph01(active, since 4h), standbys: ceph03, ceph02 osd: 55 osds: 55 up (since 4h), 54 in (since 4h); 10 remapped pgs data: pools: 3 pools, 2049 pgs objects: 8.17M objects, 29 TiB usage: 89 TiB used, 45 TiB / 133 TiB avail pgs: 27740/24497208 objects misplaced (0.113%) 2034 active+clean 9 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+remapped+backfill_wait io: recovery: 172 MiB/s, 45 objects/s root@ceph01:~# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.017034", "last_optimize_started": "Thu Jun 4 15:26:28 2020", "mode": "upmap", "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 111.70750 - 133 TiB 89 TiB 88 TiB 4.9 GiB 240 GiB 44 TiB 66.79 1.00 - root default -2 25.44400 - 25 TiB 19 TiB 19 TiB 1.0 GiB 49 GiB 6.1 TiB 75.99 1.14 - host ceph01 0 hdd 2.39999 1.00000 2.7 TiB 1.9 TiB 1.8 TiB 107 MiB 5.2 GiB 835 GiB 69.45 1.04 130 up osd.0 1 hdd 2.29999 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 79 MiB 5.4 GiB 920 GiB 66.36 0.99 124 up osd.1 4 hdd 2.67029 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 127 MiB 5.6 GiB 620 GiB 77.31 1.16 144 up osd.4 8 hdd 2.39999 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 111 MiB 5.3 GiB 755 GiB 72.38 1.08 135 up osd.8 11 hdd 1.71660 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 56 MiB 2.6 GiB 418 GiB 76.22 1.14 92 up osd.11 12 hdd 2.67029 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 122 MiB 5.3 GiB 625 GiB 77.14 1.16 144 up osd.12 14 hdd 2.67029 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 114 MiB 5.6 GiB 642 GiB 76.53 1.15 144 up osd.14 18 hdd 2.79999 1.00000 2.7 TiB 2.3 TiB 2.3 TiB 143 MiB 6.0 GiB 411 GiB 84.95 1.27 158 up osd.18 22 hdd 1.71660 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 59 MiB 2.4 GiB 388 GiB 77.94 1.17 94 up osd.22 30 hdd 1.79999 0.95001 1.7 TiB 1.4 TiB 1.4 TiB 71 MiB 2.9 GiB 324 GiB 81.58 1.22 96 up osd.30 32 hdd 0.50000 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 up osd.32 33 hdd 1.79999 0.95001 1.6 TiB 1.3 TiB 1.3 TiB 54 MiB 2.4 GiB 323 GiB 80.55 1.21 91 up osd.33 -3 24.52742 - 26 TiB 19 TiB 19 TiB 1.1 GiB 54 GiB 6.8 TiB 74.10 1.11 - host ceph02 2 hdd 1.00000 1.00000 1.7 TiB 827 GiB 825 GiB 29 MiB 2.2 GiB 931 GiB 47.05 0.70 56 up osd.2 3 hdd 2.89999 0.95001 2.7 TiB 2.2 TiB 2.2 TiB 123 MiB 6.1 GiB 454 GiB 83.38 1.25 154 up osd.3 7 hdd 2.67029 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 152 MiB 5.8 GiB 540 GiB 80.25 1.20 150 up osd.7 9 hdd 2.67029 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 172 MiB 6.0 GiB 553 GiB 79.79 1.19 149 up osd.9 13 hdd 1.79999 1.00000 2.4 TiB 1.4 TiB 1.4 TiB 70 MiB 4.6 GiB 979 GiB 59.91 0.90 100 up osd.13 16 hdd 2.89999 0.95001 2.7 TiB 2.2 TiB 2.2 TiB 140 MiB 6.0 GiB 479 GiB 82.46 1.23 153 up osd.16 19 hdd 1.39999 1.00000 1.7 TiB 1.1 TiB 1.1 TiB 41 MiB 2.8 GiB 616 GiB 64.96 0.97 78 up osd.19 23 hdd 2.00000 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 96 MiB 5.3 GiB 1.1 TiB 59.58 0.89 111 up osd.23 24 hdd 1.71660 1.00000 1.7 TiB 1.4 TiB 1.4 TiB 52 MiB 3.3 GiB 334 GiB 81.02 1.21 97 up osd.24 28 hdd 2.79999 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 143 MiB 6.2 GiB 440 GiB 83.90 1.26 155 up osd.28 31 hdd 2.67029 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 116 MiB 6.0 GiB 523 GiB 80.87 1.21 149 up osd.31 -4 20.40346 - 25 TiB 16 TiB 16 TiB 932 MiB 46 GiB 8.2 TiB 66.48 1.00 - host ceph03 5 hdd 1.71660 1.00000 1.7 TiB 1.4 TiB 1.4 TiB 61 MiB 2.8 GiB 363 GiB 79.37 1.19 95 up osd.5 6 hdd 1.71660 1.00000 1.7 TiB 1.4 TiB 1.4 TiB 67 MiB 2.9 GiB 329 GiB 81.26 1.22 97 up osd.6 10 hdd 2.50000 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 123 MiB 5.9 GiB 714 GiB 73.89 1.11 139 up osd.10 15 hdd 2.29999 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 102 MiB 5.7 GiB 860 GiB 68.55 1.03 127 up osd.15 17 hdd 1.39999 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 MiB 2.5 GiB 524 GiB 68.42 1.02 78 up osd.17 20 hdd 1.59999 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 52 MiB 3.0 GiB 444 GiB 74.73 1.12 90 up osd.20 21 hdd 2.00000 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 103 MiB 5.4 GiB 1.1 TiB 59.96 0.90 112 up osd.21 25 hdd 1.00000 1.00000 1.7 TiB 847 GiB 845 GiB 45 MiB 2.2 GiB 911 GiB 48.20 0.72 57 up osd.25 26 hdd 2.50000 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 126 MiB 5.7 GiB 719 GiB 73.70 1.10 138 up osd.26 27 hdd 1.00000 1.00000 2.7 TiB 820 GiB 816 GiB 61 MiB 3.6 GiB 1.9 TiB 29.97 0.45 56 up osd.27 29 hdd 2.67029 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 145 MiB 6.0 GiB 540 GiB 80.24 1.20 149 up osd.29 -11 12.96458 - 20 TiB 11 TiB 11 TiB 581 MiB 28 GiB 9.0 TiB 54.04 0.81 - host ceph04 34 hdd 1.00000 1.00000 2.7 TiB 819 GiB 815 GiB 70 MiB 3.6 GiB 1.9 TiB 29.96 0.45 55 up osd.34 35 hdd 1.00000 1.00000 2.7 TiB 860 GiB 857 GiB 59 MiB 3.3 GiB 1.8 TiB 31.46 0.47 57 up osd.35 44 hdd 1.00000 1.00000 2.7 TiB 856 GiB 853 GiB 56 MiB 3.6 GiB 1.8 TiB 31.31 0.47 57 up osd.44 45 hdd 1.00000 1.00000 2.7 TiB 816 GiB 812 GiB 65 MiB 3.6 GiB 1.9 TiB 29.84 0.45 55 up osd.45 48 hdd 1.71660 1.00000 1.7 TiB 1.4 TiB 1.4 TiB 50 MiB 2.7 GiB 330 GiB 81.21 1.22 95 up osd.48 51 hdd 3.62399 1.00000 3.6 TiB 3.0 TiB 2.9 TiB 143 MiB 5.3 GiB 685 GiB 81.55 1.22 201 up osd.51 52 hdd 3.62399 1.00000 3.6 TiB 3.0 TiB 3.0 TiB 138 MiB 5.4 GiB 644 GiB 82.64 1.24 204 up osd.52 -13 11.24799 - 16 TiB 9.2 TiB 9.2 TiB 490 MiB 22 GiB 6.6 TiB 58.02 0.87 - host ceph05 46 hdd 1.00000 1.00000 2.6 TiB 856 GiB 853 GiB 56 MiB 3.1 GiB 1.7 TiB 32.48 0.49 57 up osd.46 47 hdd 1.00000 1.00000 2.6 TiB 857 GiB 854 GiB 86 MiB 3.4 GiB 1.7 TiB 32.51 0.49 57 up osd.47 49 hdd 1.00000 1.00000 1.7 TiB 830 GiB 829 GiB 36 MiB 1.9 GiB 927 GiB 47.24 0.71 56 up osd.49 50 hdd 1.00000 1.00000 1.7 TiB 826 GiB 824 GiB 30 MiB 1.8 GiB 932 GiB 47.00 0.70 55 up osd.50 53 hdd 3.62399 1.00000 3.6 TiB 2.9 TiB 2.9 TiB 159 MiB 5.9 GiB 696 GiB 81.25 1.22 200 up osd.53 54 hdd 3.62399 1.00000 3.6 TiB 2.9 TiB 2.9 TiB 123 MiB 5.7 GiB 691 GiB 81.38 1.22 200 up osd.54 -9 17.12006 - 21 TiB 14 TiB 14 TiB 843 MiB 42 GiB 7.3 TiB 65.39 0.98 - host ceph06 36 hdd 2.50000 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 125 MiB 5.8 GiB 677 GiB 75.24 1.13 140 up osd.36 37 hdd 2.39999 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 120 MiB 5.8 GiB 769 GiB 71.89 1.08 134 up osd.37 38 hdd 2.67029 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 125 MiB 5.9 GiB 551 GiB 79.85 1.20 149 up osd.38 39 hdd 2.39999 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 121 MiB 5.6 GiB 773 GiB 71.74 1.07 134 up osd.39 40 hdd 2.57489 1.00000 2.6 TiB 2.0 TiB 2.0 TiB 91 MiB 5.6 GiB 543 GiB 79.40 1.19 143 up osd.40 41 hdd 1.00000 1.00000 2.6 TiB 850 GiB 846 GiB 85 MiB 3.9 GiB 1.7 TiB 32.23 0.48 57 up osd.41 42 hdd 2.57489 1.00000 2.6 TiB 2.0 TiB 2.0 TiB 90 MiB 5.6 GiB 544 GiB 79.38 1.19 142 up osd.42 43 hdd 1.00000 1.00000 2.6 TiB 844 GiB 840 GiB 87 MiB 3.9 GiB 1.8 TiB 31.99 0.48 57 up osd.43 TOTAL 133 TiB 89 TiB 88 TiB 4.9 GiB 241 GiB 45 TiB 66.79 MIN/MAX VAR: 0.45/1.27 STDDEV: 18.42

2 1

nfs-ganesha mount hangs every day since upgrade to nautilus
by Marc Roos 04 Jun '20

04 Jun '20

After having to revert back to ceph-fuse upgrading to nautilus, I have also that the nfs-ganesha mount stalls/breaks every day. Probably caused by: 1 clients failing to respond to capability release 2 clients failing to respond to cache pressure 1 MDSs report slow requests How to fix this?

1 0

speed up individual backfills
by Thomas Bennett 04 Jun '20

04 Jun '20

Hi, I have 15628 misplaced objects that are currently backfilling as follows: 1. pgid:14.3ce1 from:osd.1321 to:osd.3313 2. pgid:14.4dd9 from:osd.1693 to:osd.2980 3. pgid:14.680b from:osd.362 to:osd.3313 These are remnant backfills from a pg-upmap/rebalance campaign after we've added 2 new racks worth of osds to our cluster. Our mon db is bloated so I'm wanting to trim the mon db before continuing the next pg-upmap/rebalance campaign. So, my question is: Is there any way I can speed up the backfill process on these individual osds? Or hints to trace out why these are so slow? Regards

1 1

ceph orch upgrade stuck at the beginning.
by Gencer W. Genç 04 Jun '20

04 Jun '20

Hi, I've 15.2.1 installed on all machines. On primary machine I executed ceph upgrade command: $ ceph orch upgrade start --ceph-version 15.2.2 When I check ceph -s I see this: progress: Upgrade to docker.io/ceph/ceph:v15.2.2 (30m) [=...........................] (remaining: 8h) It says 8 hours. It is already ran for 3 hours. No upgrade processed. It get stuck at this point. Is there any way to know why this has stuck? Thanks, Gencer.

3 19

Cephadm Setup Query
by Shivanshi . 04 Jun '20

04 Jun '20

2 1

Octopus 15.2.2 unable to make drives available (reject reason locked)...
by Marco Pizzolo 04 Jun '20

04 Jun '20

Hello, Hitting an issue with a new 15.2.2 deployment using cephadm. I am having a problem creating encrypted, 2 osds per device OSDs (they are NVMe). After removing and bootstrapping the cluster again, i am unable to create OSDs as they're locked. sgdisk, wipefs, zap all fail to leave the drives as available. Any help would be appreciated. Any comments on performance experiences with ceph in containers (cephadm deployed) vs bare metal (ceph-deploy) would be greatly appreciated as well. Thanks, Marco ceph orch device ls HOST PATH TYPE SIZE DEVICE AVAIL REJECT REASONS prdhcistonode01 /dev/nvme0n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_2006266528D1 False *locked* prdhcistonode01 /dev/nvme1n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_2006266534D9 False *locked* prdhcistonode01 /dev/nvme2n1 ssd 953G INTEL SSDPEKKF010T8_BTHH850215GA1P0E False *locked* prdhcistonode01 /dev/nvme3n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_200626651473 False *locked* prdhcistonode01 /dev/nvme4n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_2006266508FB False * locked* prdhcistonode01 /dev/nvme5n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_20062664E6E8 False *locked* prdhcistonode01 /dev/nvme6n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_200626653CC0 False * locked* prdhcistonode01 /dev/nvme7n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_1939243B797E False * locked* prdhcistonode01 /dev/nvme8n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_200626652441 False *locked* lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme2n1 259:0 0 953.9G 0 disk ├─nvme2n1p1 259:1 0 512M 0 part /boot/efi └─nvme2n1p2 259:2 0 953.4G 0 part / nvme3n1 259:3 0 11.7T 0 disk └─ceph--5bd47cae--97b3--4cad--b010--215fd982497b-osd--data--e6045acd--a56d--41d2--a016--b8647b9a717a 253:1 0 11.7T 0 lvm nvme4n1 259:4 0 11.7T 0 disk └─ceph--bf7dbfb4--afe3--4391--9847--08e461bf6247-osd--data--12faafac--b695--4c30--b6d7--7046d8275d9f 253:0 0 11.7T 0 lvm nvme0n1 259:5 0 11.7T 0 disk └─ceph--1a5d8e23--ff7d--44c3--b6d2--de143fed2b7d-osd--block--b6593547--e99a--4add--8edd--5d0fb53254cd 253:2 0 11.7T 0 lvm nvme5n1 259:6 0 11.7T 0 disk └─ceph--7d85ff24--79c8--4792--a2c8--bb4908f77ff0-osd--data--fc4e9dbd--920f--41b8--8467--74e9dcbd57ca 253:3 0 11.7T 0 lvm nvme6n1 259:7 0 11.7T 0 disk └─ceph--d8c8652a--1cd8--4e10--a333--4ea10f3b5004-osd--data--9a70a549--3cba--4f0d--a13a--8465781a10e9 253:5 0 11.7T 0 lvm nvme8n1 259:8 0 11.7T 0 disk └─ceph--e1914f1c--2385--4c0c--9951--d4b9200b7164-osd--data--8876559c--6393--4fbc--821b--7ac74cfb5a54 253:7 0 11.7T 0 lvm nvme7n1 259:9 0 11.7T 0 disk └─ceph--3765b53a--75eb--489e--97e1--d6b03bc25532-osd--data--777638e0--a325--401d--a01d--459676871003 253:4 0 11.7T 0 lvm nvme1n1 259:10 0 11.7T 0 disk └─ceph--2124f206--2b50--41a1--8a3c--d47c1a909a3b-osd--block--88e4f1eb--73f4--4c83--b978--fe7cabc0c3e6 253:6 0 11.7T 0 lvm

2 2

15.2.3 Crush Map Viewer problem.
by Marco Pizzolo 04 Jun '20

04 Jun '20

Hello Everyone, We're working on a new cluster and seeing some oddities. The crush map viewer is not showing all hosts or OSDs. Cluster is NVMe w/4 hosts, each having 8 NVMe. Using 2 OSDs per NVMe and Encryption. Using Max size of 3, Min size of 2: [image: image.png] All OSDs appear to exist in: ceph osd tree root@prdhcistonode01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 372.60156 root default -3 93.15039 host prdhcistonode01 0 ssd 5.82190 osd.0 up 1.00000 1.00000 1 ssd 5.82190 osd.1 up 1.00000 1.00000 2 ssd 5.82190 osd.2 up 1.00000 1.00000 3 ssd 5.82190 osd.3 up 1.00000 1.00000 4 ssd 5.82190 osd.4 up 1.00000 1.00000 5 ssd 5.82190 osd.5 up 1.00000 1.00000 6 ssd 5.82190 osd.6 up 1.00000 1.00000 7 ssd 5.82190 osd.7 up 1.00000 1.00000 8 ssd 5.82190 osd.8 up 1.00000 1.00000 9 ssd 5.82190 osd.9 up 1.00000 1.00000 10 ssd 5.82190 osd.10 up 1.00000 1.00000 11 ssd 5.82190 osd.11 up 1.00000 1.00000 12 ssd 5.82190 osd.12 up 1.00000 1.00000 13 ssd 5.82190 osd.13 up 1.00000 1.00000 14 ssd 5.82190 osd.14 up 1.00000 1.00000 15 ssd 5.82190 osd.15 up 1.00000 1.00000 -5 93.15039 host prdhcistonode02 17 ssd 5.82190 osd.17 up 1.00000 1.00000 18 ssd 5.82190 osd.18 up 1.00000 1.00000 19 ssd 5.82190 osd.19 up 1.00000 1.00000 20 ssd 5.82190 osd.20 up 1.00000 1.00000 21 ssd 5.82190 osd.21 up 1.00000 1.00000 22 ssd 5.82190 osd.22 up 1.00000 1.00000 23 ssd 5.82190 osd.23 up 1.00000 1.00000 24 ssd 5.82190 osd.24 up 1.00000 1.00000 25 ssd 5.82190 osd.25 up 1.00000 1.00000 26 ssd 5.82190 osd.26 up 1.00000 1.00000 27 ssd 5.82190 osd.27 up 1.00000 1.00000 28 ssd 5.82190 osd.28 up 1.00000 1.00000 29 ssd 5.82190 osd.29 up 1.00000 1.00000 30 ssd 5.82190 osd.30 up 1.00000 1.00000 48 ssd 5.82190 osd.48 up 1.00000 1.00000 49 ssd 5.82190 osd.49 up 1.00000 1.00000 -7 93.15039 host prdhcistonode03 16 ssd 5.82190 osd.16 up 1.00000 1.00000 31 ssd 5.82190 osd.31 up 1.00000 1.00000 32 ssd 5.82190 osd.32 up 1.00000 1.00000 33 ssd 5.82190 osd.33 up 1.00000 1.00000 34 ssd 5.82190 osd.34 up 1.00000 1.00000 35 ssd 5.82190 osd.35 up 1.00000 1.00000 36 ssd 5.82190 osd.36 up 1.00000 1.00000 37 ssd 5.82190 osd.37 up 1.00000 1.00000 38 ssd 5.82190 osd.38 up 1.00000 1.00000 39 ssd 5.82190 osd.39 up 1.00000 1.00000 40 ssd 5.82190 osd.40 up 1.00000 1.00000 41 ssd 5.82190 osd.41 up 1.00000 1.00000 42 ssd 5.82190 osd.42 up 1.00000 1.00000 43 ssd 5.82190 osd.43 up 1.00000 1.00000 44 ssd 5.82190 osd.44 up 1.00000 1.00000 45 ssd 5.82190 osd.45 up 1.00000 1.00000 -9 93.15039 host prdhcistonode04 46 ssd 5.82190 osd.46 up 1.00000 1.00000 47 ssd 5.82190 osd.47 up 1.00000 1.00000 50 ssd 5.82190 osd.50 up 1.00000 1.00000 51 ssd 5.82190 osd.51 up 1.00000 1.00000 52 ssd 5.82190 osd.52 up 1.00000 1.00000 53 ssd 5.82190 osd.53 up 1.00000 1.00000 54 ssd 5.82190 osd.54 up 1.00000 1.00000 55 ssd 5.82190 osd.55 up 1.00000 1.00000 56 ssd 5.82190 osd.56 up 1.00000 1.00000 57 ssd 5.82190 osd.57 up 1.00000 1.00000 58 ssd 5.82190 osd.58 up 1.00000 1.00000 59 ssd 5.82190 osd.59 up 1.00000 1.00000 60 ssd 5.82190 osd.60 up 1.00000 1.00000 61 ssd 5.82190 osd.61 up 1.00000 1.00000 62 ssd 5.82190 osd.62 up 1.00000 1.00000 63 ssd 5.82190 osd.63 up 1.00000 1.00000 Any suggestions would be greatly appreciated. Thanks, Marco

2 3

RBD logs
by 陈旭 04 Jun '20

04 Jun '20

Hi guys, I deploy an efk cluster and use ceph as block storage in kubernetes, but RBD write iops sometimes becomes zero and last for a few minutes. I want to check logs about RBD so I add some config to ceph.conf and restart ceph. Here is my ceph.conf: [global] fsid = 53f4e1d5-32ce-4e9c-bf36-f6b54b009962 mon_initial_members = db-16-4-hzxs, db-16-5-hzxs, db-16-6-hzxs mon_host = 10.25.16.4,10.25.16.5,10.25.16.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 3 [client] debug rbd = 20 debug rbd mirror = 20 debug rbd replay = 20 log file = /var/log/ceph/client_rbd.log I can not get any logs in /var/log/ceph/client_rbd.log. I also try to execute 'ceph daemon osd.* config set debug_rbd 20’ and there is also no related logs in ceph-osd.log. How can I get useful logs about this question or How can I analyze this problem? Look forward to your reply. Thanks ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也不接受任何延迟收件的赔偿责任。 ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notice: This email may contain confidential or privileged information of Yitu and was sent solely to the intended recipients. If you are unauthorized to receive this email, you should delete the email and contact the sender immediately. Any unauthorized disclosing, distribution, or copying of this email and attachment thereto is prohibited. Yitu does not accept any liability for any loss caused by possibly viruses in this email. E-mail transmission cannot be guaranteed to be secure or error-free and Yitu is not responsible for any delayed transmission.

2 1

Help! ceph-mon is blocked after shutting down and ip address changed
by occj＠qq.com 04 Jun '20

04 Jun '20

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) os :CentOS Linux release 7.7.1908 (Core) single node ceph cluster with 1 mon,1mgr,1 mds,1rgw and 12osds , but only cephfs is used. ceph -s is blocked after shutting down the machine (192.168.0.104), then ip address changed to 192.168.1.6 I created the monmap with monmap tool and update the ceph.conf , hosts file and then start ceph-mon. and the ceph-mon log: ... 2019-12-11 08:57:45.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1285.14s 2019-12-11 08:57:50.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1290.14s 2019-12-11 08:57:55.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1295.14s 2019-12-11 08:58:00.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1300.14s 2019-12-11 08:58:05.172 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1305.14s 2019-12-11 08:58:10.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1310.14s 2019-12-11 08:58:15.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1315.14s 2019-12-11 08:58:20.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1320.14s 2019-12-11 08:58:25.174 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1325.14s ... I changed IP back to 192.168.0.104 yeasterday, but all the same. # cat /etc/ceph/ceph.conf [client.libvirt] admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor log file = /var/log/ceph/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor [client.rgw.ceph-node1.rgw0] host = ceph-node1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-node1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-node1.rgw0.log rgw frontends = beast endpoint=192.168.1.6:8080 rgw thread pool size = 512 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.1.0/24 fsid = e384e8e6-94d5-4812-bfbb-d1b0468bdef5 mon host = [v2:192.168.1.6:3300,v1:192.168.1.6:6789] mon initial members = ceph-node1 osd crush chooseleaf type = 0 osd pool default crush rule = -1 public network = 192.168.1.0/24 [osd] osd memory target = 7870655146

2 1