- ceph-users - lists.ceph.io

Ceph Octopus packages missing at download.ceph.com
by bzieglmeier＠gmail.com 18 Nov '24

18 Nov '24

Today I noticed that all ceph octopus packages are missing from download.ceph.com. Is this intentional? Was this an accident? I'm unable to find any announcement or existing issue tracking this...

6 6

Stray monitor
by Jakub Daniel 18 Nov '24

18 Nov '24

Hello, I'm pretty new to ceph deployment. I have setup my first cephfs cluster using cephadm. Initially, I deployed ceph in 3 virtualbox instances that I called cephfs-cluster-node-{0, 1, 2} just to test things. Later, I added 5 more real hardware nodes. Later I decided I'd remove the virtualboxes, so I drained the osds and removed the hosts. Suddenly, ceph status detail started reporting HEALTH_WARN 1 stray host(s) with 1 daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by cephadm stray host cephfs-cluster-node-2 has 1 stray daemons: ['mon.X'] The cephfs-cluster-node-2 is no longer listed among hosts, it is (and has been for tens of hours) offline (powered down). The mon.X doesn't even belong to that node, it is one of the real hardware nodes. I am unaware of mon.X ever running on cephfs-cluster-node-2 (never noticed it among systemd units). Where does cephadm shell -- ceph status detail come to the conclusion there is something stray? How can I address this? Thank you for any insights Jakub

3 5

done, waiting for purge
by Torkil Svensgaard 18 Nov '24

18 Nov '24

Hi 18.2.4 We had some hard drives going AWOL due to a failing SAS expander so I initiated "ceph orch host drain host". After a couple days I'm now looking at this: " OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 528 gimpy done, waiting for purge 0 False False False 529 gimpy done, waiting for purge 0 False False False 530 gimpy done, waiting for purge 0 False False False 531 gimpy done, waiting for purge 0 False False False 532 gimpy done, waiting for purge 0 False False False 533 gimpy done, waiting for purge 0 False False False 534 gimpy done, waiting for purge 0 False False False 535 gimpy done, waiting for purge 0 False False False 536 gimpy done, waiting for purge 0 False False False 537 gimpy done, waiting for purge 0 False False False 538 gimpy done, waiting for purge 0 False False False 539 gimpy done, waiting for purge 0 False False False " It removed the drives that was still working just fine but the missing drives seem stuck like this. How do I get these to finish? Force rm? Mvh. Torkil

2 1

Re: The effect of changing an osd's class
by Roland Giesler 18 Nov '24

18 Nov '24

On 2024/11/16 18:38, Anthony D'Atri wrote: > Disabling mclock as described here > https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/ might > help I cannot see any option that allows me to disable mclock... > > Also, you have a small cluster with a bunch of small OSDs. Please > send `ceph health detail` # ceph health detail HEALTH_WARN 1 pool(s) have no replicas configured [WRN] POOL_NO_REDUNDANCY: 1 pool(s) have no replicas configured pool 'backups' has no replicas configured The backups pool is on spinners and 1 copy is sufficient since we're also replicating this offsite in real time. Otherwise too many copies take too much space. > > Please also send `ceph osd df` and `ceph osd crush dump` NodeC:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 hdd 1.86029 1.00000 1.9 TiB 1.1 TiB 1.1 TiB 16 MiB 2.7 GiB 746 GiB 60.97 1.23 68 up 3 hdd 1.86029 1.00000 1.9 TiB 949 GiB 902 GiB 12 MiB 2.6 GiB 961 GiB 49.68 1.00 57 up 4 hdd 1.86029 1.00000 1.9 TiB 848 GiB 800 GiB 10 MiB 2.3 GiB 1.0 TiB 44.37 0.89 51 up 5 hdd 1.86589 1.00000 1.9 TiB 1.0 TiB 982 GiB 15 MiB 2.4 GiB 881 GiB 53.87 1.08 66 up 0 ssd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 1 ssd 0.28319 1.00000 290 GiB 123 GiB 122 GiB 2.3 MiB 1.2 GiB 167 GiB 42.38 0.85 46 up 36 ssd 0 1.00000 290 GiB 1.3 GiB 23 MiB 2.8 MiB 1.2 GiB 289 GiB 0.44 0.01 13 up 37 ssd 0 1.00000 290 GiB 1.1 GiB 23 MiB 2.5 MiB 1.1 GiB 289 GiB 0.38 0.01 18 up 38 ssd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 41 up 39 ssd 0 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 86 up 40 ssd 3.30690 1.00000 3.3 TiB 1.7 TiB 1.7 TiB 23 MiB 4.8 GiB 1.6 TiB 50.49 1.02 424 up 10 hdd 1.86029 1.00000 1.9 TiB 1.0 TiB 990 GiB 13 MiB 2.8 GiB 873 GiB 54.19 1.09 64 up 11 hdd 1.86029 1.00000 1.9 TiB 1.3 TiB 1.2 TiB 15 MiB 3.2 GiB 625 GiB 67.20 1.35 72 up 26 hdd 1.86029 1.00000 1.9 TiB 843 GiB 801 GiB 12 MiB 2.3 GiB 1.0 TiB 44.25 0.89 50 up 27 hdd 1.86029 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 16 MiB 2.7 GiB 793 GiB 58.38 1.18 70 up 6 ssd 0.28319 1.00000 290 GiB 115 GiB 115 GiB 2.3 MiB 743 MiB 175 GiB 39.82 0.80 28 up 7 ssd 0.28319 1.00000 290 GiB 94 GiB 93 GiB 2.0 MiB 921 MiB 196 GiB 32.51 0.65 23 up 8 ssd 0.28319 1.00000 290 GiB 103 GiB 102 GiB 2.1 MiB 963 MiB 187 GiB 35.46 0.71 27 up 9 ssd 0.28319 1.00000 290 GiB 131 GiB 130 GiB 2.5 MiB 1.1 GiB 159 GiB 45.28 0.91 33 up 24 ssd 0.28319 1.00000 290 GiB 102 GiB 101 GiB 2.1 MiB 719 MiB 188 GiB 35.10 0.71 26 up 25 ssd 0.28319 1.00000 290 GiB 152 GiB 151 GiB 2.8 MiB 1020 MiB 138 GiB 52.52 1.06 39 up 41 ssd 3.49309 1.00000 3.5 TiB 1.6 TiB 1.6 TiB 22 MiB 5.4 GiB 1.9 TiB 44.58 0.90 411 up 14 hdd 1.59999 1.00000 1.9 TiB 1.0 TiB 1.0 TiB 13 MiB 2.4 GiB 837 GiB 56.04 1.13 63 up 15 hdd 1.86029 1.00000 1.9 TiB 1017 GiB 975 GiB 13 MiB 2.1 GiB 888 GiB 53.36 1.07 60 up 16 hdd 1.86029 1.00000 1.9 TiB 990 GiB 948 GiB 13 MiB 2.2 GiB 915 GiB 51.97 1.05 59 up 17 hdd 1.86029 1.00000 1.9 TiB 1.0 TiB 992 GiB 14 MiB 2.5 GiB 871 GiB 54.28 1.09 64 up 12 ssd 0.28319 1.00000 290 GiB 122 GiB 121 GiB 2.3 MiB 1.1 GiB 168 GiB 41.96 0.84 31 up 13 ssd 0.28319 1.00000 290 GiB 145 GiB 144 GiB 2.6 MiB 1.2 GiB 145 GiB 49.95 1.01 37 up 28 ssd 0.28319 1.00000 290 GiB 142 GiB 141 GiB 2.6 MiB 1.1 GiB 148 GiB 48.89 0.98 35 up 29 ssd 0.28319 1.00000 290 GiB 109 GiB 108 GiB 2.2 MiB 1.3 GiB 181 GiB 37.58 0.76 29 up 30 ssd 0.28319 1.00000 290 GiB 146 GiB 145 GiB 2.7 MiB 1.3 GiB 144 GiB 50.37 1.01 37 up 31 ssd 0.28319 1.00000 290 GiB 131 GiB 130 GiB 2.5 MiB 1.3 GiB 159 GiB 45.24 0.91 34 up 43 ssd 3.30690 1.00000 3.3 TiB 1.4 TiB 1.4 TiB 19 MiB 4.0 GiB 1.9 TiB 42.50 0.86 353 up 20 hdd 1.86029 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 12 MiB 2.8 GiB 659 GiB 65.38 1.32 67 up 21 hdd 1.86029 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 15 MiB 2.8 GiB 815 GiB 57.20 1.15 68 up 22 hdd 1.86029 1.00000 1.9 TiB 878 GiB 836 GiB 13 MiB 2.4 GiB 1.0 TiB 46.10 0.93 54 up 23 hdd 1.86029 1.00000 1.9 TiB 1018 GiB 977 GiB 14 MiB 2.8 GiB 886 GiB 53.46 1.08 59 up 18 ssd 0.28319 1.00000 290 GiB 115 GiB 114 GiB 2.3 MiB 1.3 GiB 175 GiB 39.74 0.80 29 up 19 ssd 0.28319 1.00000 290 GiB 115 GiB 114 GiB 2.3 MiB 961 MiB 175 GiB 39.59 0.80 28 up 32 ssd 0.28319 1.00000 290 GiB 103 GiB 102 GiB 2.1 MiB 1.0 GiB 187 GiB 35.65 0.72 26 up 33 ssd 0.28319 1.00000 290 GiB 98 GiB 97 GiB 2.1 MiB 1.4 GiB 192 GiB 33.92 0.68 26 up 34 ssd 0.28319 1.00000 290 GiB 141 GiB 140 GiB 2.6 MiB 987 MiB 149 GiB 48.49 0.98 35 up 35 ssd 0.28319 1.00000 290 GiB 116 GiB 114 GiB 2.3 MiB 1.3 GiB 174 GiB 39.92 0.80 31 up 42 ssd 3.49309 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 20 MiB 4.5 GiB 2.1 TiB 40.45 0.81 372 up TOTAL 49 TiB 25 TiB 24 TiB 351 MiB 83 GiB 25 TiB 49.67 MIN/MAX VAR: 0.01/1.35 STDDEV: 13.87 NodeC:~# ceph osd crush dump { "devices": [ { "id": 0, "name": "osd.0", "class": "ssd" }, { "id": 1, "name": "osd.1", "class": "ssd" }, { "id": 2, "name": "osd.2", "class": "hdd" }, { "id": 3, "name": "osd.3", "class": "hdd" }, { "id": 4, "name": "osd.4", "class": "hdd" }, { "id": 5, "name": "osd.5", "class": "hdd" }, { "id": 6, "name": "osd.6", "class": "ssd" }, { "id": 7, "name": "osd.7", "class": "ssd" }, { "id": 8, "name": "osd.8", "class": "ssd" }, { "id": 9, "name": "osd.9", "class": "ssd" }, { "id": 10, "name": "osd.10", "class": "hdd" }, { "id": 11, "name": "osd.11", "class": "hdd" }, { "id": 12, "name": "osd.12", "class": "ssd" }, { "id": 13, "name": "osd.13", "class": "ssd" }, { "id": 14, "name": "osd.14", "class": "hdd" }, { "id": 15, "name": "osd.15", "class": "hdd" }, { "id": 16, "name": "osd.16", "class": "hdd" }, { "id": 17, "name": "osd.17", "class": "hdd" }, { "id": 18, "name": "osd.18", "class": "ssd" }, { "id": 19, "name": "osd.19", "class": "ssd" }, { "id": 20, "name": "osd.20", "class": "hdd" }, { "id": 21, "name": "osd.21", "class": "hdd" }, { "id": 22, "name": "osd.22", "class": "hdd" }, { "id": 23, "name": "osd.23", "class": "hdd" }, { "id": 24, "name": "osd.24", "class": "ssd" }, { "id": 25, "name": "osd.25", "class": "ssd" }, { "id": 26, "name": "osd.26", "class": "hdd" }, { "id": 27, "name": "osd.27", "class": "hdd" }, { "id": 28, "name": "osd.28", "class": "ssd" }, { "id": 29, "name": "osd.29", "class": "ssd" }, { "id": 30, "name": "osd.30", "class": "ssd" }, { "id": 31, "name": "osd.31", "class": "ssd" }, { "id": 32, "name": "osd.32", "class": "ssd" }, { "id": 33, "name": "osd.33", "class": "ssd" }, { "id": 34, "name": "osd.34", "class": "ssd" }, { "id": 35, "name": "osd.35", "class": "ssd" }, { "id": 36, "name": "osd.36", "class": "ssd" }, { "id": 37, "name": "osd.37", "class": "ssd" }, { "id": 38, "name": "osd.38", "class": "ssd" }, { "id": 39, "name": "osd.39", "class": "ssd" }, { "id": 40, "name": "osd.40", "class": "ssd" }, { "id": 41, "name": "osd.41", "class": "ssd" }, { "id": 42, "name": "osd.42", "class": "ssd" }, { "id": 43, "name": "osd.43", "class": "ssd" } ], "types": [ { "type_id": 0, "name": "osd" }, { "type_id": 1, "name": "host" }, { "type_id": 2, "name": "chassis" }, { "type_id": 3, "name": "rack" }, { "type_id": 4, "name": "row" }, { "type_id": 5, "name": "pdu" }, { "type_id": 6, "name": "pod" }, { "type_id": 7, "name": "room" }, { "type_id": 8, "name": "datacenter" }, { "type_id": 9, "name": "zone" }, { "type_id": 10, "name": "region" }, { "type_id": 11, "name": "root" } ], "buckets": [ { "id": -1, "name": "default", "type_id": 11, "type_name": "root", "weight": 3177873, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -3, "weight": 723311, "pos": 0 }, { "id": -7, "weight": 827941, "pos": 1 }, { "id": -10, "weight": 798680, "pos": 2 }, { "id": -13, "weight": 827941, "pos": 3 } ] }, { "id": -2, "name": "default~ssd", "type_id": 11, "type_name": "root", "weight": 1243909, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -4, "weight": 235280, "pos": 0 }, { "id": -8, "weight": 340277, "pos": 1 }, { "id": -11, "weight": 328075, "pos": 2 }, { "id": -14, "weight": 340277, "pos": 3 } ] }, { "id": -3, "name": "FT1-NodeA", "type_id": 1, "type_name": "host", "weight": 723311, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 1, "weight": 18559, "pos": 0 }, { "id": 36, "weight": 0, "pos": 1 }, { "id": 37, "weight": 0, "pos": 2 }, { "id": 0, "weight": 0, "pos": 3 }, { "id": 38, "weight": 0, "pos": 4 }, { "id": 39, "weight": 0, "pos": 5 }, { "id": 2, "weight": 121916, "pos": 6 }, { "id": 3, "weight": 121916, "pos": 7 }, { "id": 4, "weight": 121916, "pos": 8 }, { "id": 40, "weight": 216721, "pos": 9 }, { "id": 5, "weight": 122283, "pos": 10 } ] }, { "id": -4, "name": "FT1-NodeA~ssd", "type_id": 1, "type_name": "host", "weight": 235280, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 1, "weight": 18559, "pos": 0 }, { "id": 36, "weight": 0, "pos": 1 }, { "id": 37, "weight": 0, "pos": 2 }, { "id": 0, "weight": 0, "pos": 3 }, { "id": 38, "weight": 0, "pos": 4 }, { "id": 39, "weight": 0, "pos": 5 }, { "id": 40, "weight": 216721, "pos": 6 } ] }, { "id": -5, "name": "FT1-NodeA~hdd", "type_id": 1, "type_name": "host", "weight": 488031, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 121916, "pos": 0 }, { "id": 3, "weight": 121916, "pos": 1 }, { "id": 4, "weight": 121916, "pos": 2 }, { "id": 5, "weight": 122283, "pos": 3 } ] }, { "id": -6, "name": "default~hdd", "type_id": 11, "type_name": "root", "weight": 1933964, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -5, "weight": 488031, "pos": 0 }, { "id": -9, "weight": 487664, "pos": 1 }, { "id": -12, "weight": 470605, "pos": 2 }, { "id": -15, "weight": 487664, "pos": 3 } ] }, { "id": -7, "name": "FT1-NodeB", "type_id": 1, "type_name": "host", "weight": 827941, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 6, "weight": 18559, "pos": 0 }, { "id": 24, "weight": 18559, "pos": 1 }, { "id": 25, "weight": 18559, "pos": 2 }, { "id": 11, "weight": 121916, "pos": 3 }, { "id": 10, "weight": 121916, "pos": 4 }, { "id": 7, "weight": 18559, "pos": 5 }, { "id": 8, "weight": 18559, "pos": 6 }, { "id": 9, "weight": 18559, "pos": 7 }, { "id": 26, "weight": 121916, "pos": 8 }, { "id": 27, "weight": 121916, "pos": 9 }, { "id": 41, "weight": 228923, "pos": 10 } ] }, { "id": -8, "name": "FT1-NodeB~ssd", "type_id": 1, "type_name": "host", "weight": 340277, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 6, "weight": 18559, "pos": 0 }, { "id": 24, "weight": 18559, "pos": 1 }, { "id": 25, "weight": 18559, "pos": 2 }, { "id": 7, "weight": 18559, "pos": 3 }, { "id": 8, "weight": 18559, "pos": 4 }, { "id": 9, "weight": 18559, "pos": 5 }, { "id": 41, "weight": 228923, "pos": 6 } ] }, { "id": -9, "name": "FT1-NodeB~hdd", "type_id": 1, "type_name": "host", "weight": 487664, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 11, "weight": 121916, "pos": 0 }, { "id": 10, "weight": 121916, "pos": 1 }, { "id": 26, "weight": 121916, "pos": 2 }, { "id": 27, "weight": 121916, "pos": 3 } ] }, { "id": -10, "name": "FT1-NodeC", "type_id": 1, "type_name": "host", "weight": 798680, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 13, "weight": 18559, "pos": 0 }, { "id": 28, "weight": 18559, "pos": 1 }, { "id": 29, "weight": 18559, "pos": 2 }, { "id": 14, "weight": 104857, "pos": 3 }, { "id": 12, "weight": 18559, "pos": 4 }, { "id": 30, "weight": 18559, "pos": 5 }, { "id": 31, "weight": 18559, "pos": 6 }, { "id": 15, "weight": 121916, "pos": 7 }, { "id": 16, "weight": 121916, "pos": 8 }, { "id": 17, "weight": 121916, "pos": 9 }, { "id": 43, "weight": 216721, "pos": 10 } ] }, { "id": -11, "name": "FT1-NodeC~ssd", "type_id": 1, "type_name": "host", "weight": 328075, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 13, "weight": 18559, "pos": 0 }, { "id": 28, "weight": 18559, "pos": 1 }, { "id": 29, "weight": 18559, "pos": 2 }, { "id": 12, "weight": 18559, "pos": 3 }, { "id": 30, "weight": 18559, "pos": 4 }, { "id": 31, "weight": 18559, "pos": 5 }, { "id": 43, "weight": 216721, "pos": 6 } ] }, { "id": -12, "name": "FT1-NodeC~hdd", "type_id": 1, "type_name": "host", "weight": 470605, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 14, "weight": 104857, "pos": 0 }, { "id": 15, "weight": 121916, "pos": 1 }, { "id": 16, "weight": 121916, "pos": 2 }, { "id": 17, "weight": 121916, "pos": 3 } ] }, { "id": -13, "name": "FT1-NodeD", "type_id": 1, "type_name": "host", "weight": 827941, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 18, "weight": 18559, "pos": 0 }, { "id": 32, "weight": 18559, "pos": 1 }, { "id": 33, "weight": 18559, "pos": 2 }, { "id": 22, "weight": 121916, "pos": 3 }, { "id": 19, "weight": 18559, "pos": 4 }, { "id": 34, "weight": 18559, "pos": 5 }, { "id": 35, "weight": 18559, "pos": 6 }, { "id": 23, "weight": 121916, "pos": 7 }, { "id": 21, "weight": 121916, "pos": 8 }, { "id": 20, "weight": 121916, "pos": 9 }, { "id": 42, "weight": 228923, "pos": 10 } ] }, { "id": -14, "name": "FT1-NodeD~ssd", "type_id": 1, "type_name": "host", "weight": 340277, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 18, "weight": 18559, "pos": 0 }, { "id": 32, "weight": 18559, "pos": 1 }, { "id": 33, "weight": 18559, "pos": 2 }, { "id": 19, "weight": 18559, "pos": 3 }, { "id": 34, "weight": 18559, "pos": 4 }, { "id": 35, "weight": 18559, "pos": 5 }, { "id": 42, "weight": 228923, "pos": 6 } ] }, { "id": -15, "name": "FT1-NodeD~hdd", "type_id": 1, "type_name": "host", "weight": 487664, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 22, "weight": 121916, "pos": 0 }, { "id": 23, "weight": 121916, "pos": 1 }, { "id": 21, "weight": 121916, "pos": 2 }, { "id": 20, "weight": 121916, "pos": 3 } ] }, { "id": -16, "name": "FT1-NodeA~old-ssd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, { "id": -17, "name": "FT1-NodeB~old-ssd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, { "id": -18, "name": "FT1-NodeC~old-ssd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, { "id": -19, "name": "FT1-NodeD~old-ssd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, { "id": -20, "name": "default~old-ssd", "type_id": 11, "type_name": "root", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -16, "weight": 0, "pos": 0 }, { "id": -17, "weight": 0, "pos": 1 }, { "id": -18, "weight": 0, "pos": 2 }, { "id": -19, "weight": 0, "pos": 3 } ] } ], "rules": [ { "rule_id": 0, "rule_name": "ssd_rule", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "hdd_rule", "type": 1, "steps": [ { "op": "take", "item": -6, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "old_nvme", "type": 1, "steps": [ { "op": "take", "item": -20, "item_name": "default~old-ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ], "tunables": { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 }, "choose_args": {} } > > It may help to manually edit the CRUSH map as detailed here: > > https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/ > > > At the end of the decompiled CRUSH map is a “tunables” section. Look > for the “c|hoose_total_tries|” > line, which should have the default value of 50. Change that to 100. > > > >> On Nov 16, 2024, at 11:13 AM, Roland Giesler <roland(a)giesler.za.net> >> wrote: >> >> On 2024/11/15 16:37, Anthony D'Atri wrote: >>> Only 2 OSDs in the acting set. This isn’t a size=2 pool is it? Did >>> you restart osd.39? >> >> Restarted both repeatedly. Has no effect. >> >> >>> >>>> On Nov 15, 2024, at 9:32 AM, Roland Giesler <roland(a)giesler.za.net> >>>> wrote: >>>> >>>> "acting": [ >>>> 39, >>>> 1 >>> >

3 5

Re: Pacific: mgr loses osd removal queue
by Eugen Block 18 Nov '24

18 Nov '24

I have a feeling that this could be related to the drives, but I have no real proof. I drained the SSD OSDs yesterday, hours later I wanted to remove the OSDs (no PGs were on them anymore) via for loop with the orchestrator (ceph orch osd rm ID --force --zap). The first one got removed quite quickly, but the others disappeared from the queue. I tried again and looked at the iostat output on the node where the queued OSD was running on. The drive had no IO at all but was utilized 100% for several minutes until it was eventually removed. I find that very weird, especially since we’re currently helping a customer rebuild OSDs on Pacific as well, and I haven’t seen such a behavior yet. And we already redeployed 132 OSDs so far. Zitat von Eugen Block <eblock(a)nde.ag>: > Hi, > > I'm not sure if this has been asked before, or if there's an > existing tracker issue already. It's difficult to reproduce it on my > lab clusters. > I'm testing some new SSD OSDs on a Pacific cluster (16.2.15) and > noticed that if we instruct the orchestrator to remove two or three > OSDs (issuing the command 'ceph orch osd rm {ID}' a couple of > times), it eventually only removes the first in the queue. I've been > watching 'ceph orch osd rm status' to see the progress, and then the > rest of the queued OSDs suddenly vanish from the status and never > get removed. Then I have to issue the command again. If I remove one > OSD by one, so no others in the queue, they are all successfully > removed. Why is this happening? Is this a bug in Pacific? Has > someone seen this in newer releases? > > Thanks! > Eugen

2 2

RGW names disappeared in quincy
by Szabo, Istvan (Agoda) 18 Nov '24

18 Nov '24

Hi, After updated from octopus to quincy the rados-gateways are showing like: rgw: 35 daemons active (20 hosts, 1 zones) which wouldn't be an issue, however the metrics that are exposed are coming with instance_id only (3642302922) which is pretty difficult (impossible) to match where it is running. This id can't be seen neither in the logs, nor as a socket. (Our deployment is not containerized) Is there a way to somehow make it easier to identify or somehow match with the servers where they are running? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Question about speeding hdd based cluster
by Kyriazis, George 16 Nov '24

16 Nov '24

Hello ceph-users, I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools. What would increase the speed of the hdd-based cephfs more, and in what usage scenarios: 1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links. It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck? Thank you! George

6 15

The effect of changing an osd's class
by Roland Giesler 15 Nov '24

15 Nov '24

I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it. When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd. What happened when I did this?

6 15

Ceph Reef 16 pgs not deep scrub and scrub
by Saint Kid 15 Nov '24

15 Nov '24

Hello, I have exactly 16 pgs with these conditions. Is there anyway i could do? I have tried to initiate the scrubing both deep and normal but they remain the same. HEALTH_WARN: 16 pgs not deep-scrubbed in time pg 8.14 not deep-scrubbed since 2024-09-27T19:42:47.463766+0700 pg 7.1b not deep-scrubbed since 2024-09-30T21:54:16.584697+0700 pg 8.1b not deep-scrubbed since 2024-09-29T23:22:37.812579+0700 pg 8.1d not deep-scrubbed since 2024-10-01T03:51:01.714780+0700 pg 6.13 not deep-scrubbed since 2024-09-26T07:16:31.387575+0700 pg 6.12 not deep-scrubbed since 2024-09-27T04:55:34.699188+0700 pg 8.3 not deep-scrubbed since 2024-09-29T20:57:40.973759+0700 pg 7.c not deep-scrubbed since 2024-10-01T22:57:28.955052+0700 pg 8.b not deep-scrubbed since 2024-10-01T09:15:55.223268+0700 pg 7.3 not deep-scrubbed since 2024-10-02T05:12:50.743544+0700 pg 7.18 not deep-scrubbed since 2024-09-30T17:57:34.550318+0700 pg 8.a not deep-scrubbed since 2024-09-30T10:37:06.429820+0700 pg 8.5 not deep-scrubbed since 2024-09-25T19:23:57.422280+0700 pg 8.9 not deep-scrubbed since 2024-09-30T17:57:33.493494+0700 pg 8.4 not deep-scrubbed since 2024-09-30T21:54:13.122287+0700 pg 8.10 not deep-scrubbed since 2024-10-02T20:07:42.793192+0700 HEALTH_WARN: 16 pgs not scrubbed in time pg 8.14 not scrubbed since 2024-10-02T08:04:26.363903+0700 pg 7.1b not scrubbed since 2024-10-02T03:14:53.854041+0700 pg 8.1b not scrubbed since 2024-10-02T15:06:12.351794+0700 pg 8.1d not scrubbed since 2024-10-02T11:57:57.600594+0700 pg 6.13 not scrubbed since 2024-10-02T01:48:49.382133+0700 pg 6.12 not scrubbed since 2024-10-02T08:19:53.026422+0700 pg 8.3 not scrubbed since 2024-10-02T04:29:55.426251+0700 pg 7.c not scrubbed since 2024-10-01T22:57:28.955052+0700 pg 8.b not scrubbed since 2024-10-02T09:36:03.568661+0700 pg 7.3 not scrubbed since 2024-10-02T05:12:50.743544+0700 pg 7.18 not scrubbed since 2024-10-01T22:17:53.504919+0700 pg 8.a not scrubbed since 2024-10-01T19:42:53.953996+0700 pg 8.5 not scrubbed since 2024-10-02T10:04:33.412894+0700 pg 8.9 not scrubbed since 2024-10-01T20:22:40.181880+0700 pg 8.4 not scrubbed since 2024-10-01T21:56:08.596293+0700 pg 8.10 not scrubbed since 2024-10-02T20:07:42.793192+0700

3 2

cephadm node failure (re-use OSDs instead of reprovisioning)
by Stefan Kooman 13 Nov '24

13 Nov '24

Hi list, In case of an OS disk failure on a cephadm managed storage node, is there a way to redeploy ceph on the (reinstalled) node leaving the data (OSDs) intact? So instead of removing the storage node, have the cluster recover, redeploy the storage node, and let the cluster recover, I would like to skip both recovery steps (when all that is broken is an OS disk) and only have a little bit of recovery while the OSDs where down. With a package based install this is achieved pretty easily: reinstall OS / ceph, make sure ceph keyrings are in right place / right owner (ceph) and run: ceph-volume lvm activate --all ... and you are back in business. Thanks, Stefan

2 2