- ceph-users - lists.ceph.io

stretched cluster or not, with mon in 3 DC and osds on 2 DC
by aderumier＠odiso.com 14 Jun '21

14 Jun '21

Hi, I'm currently reading the documentation about stretched cluster, I would like to known if it's needed or not with this kind of 3 dc setup: 3km (0.2ms) DC1--------------DC2 30km(3ms) | | 30km (2-3ms) |--------DC3------ DC1 && DC2 are near each other, small latency. (0.2ms) DC3 is at 30km with bigger latency. (2ms) separated links between dc with different physical path 1 monitor on each dc osd on DC1/DC2, with size=4 Cluster is full nvme or ssd, lowest latency is required for osd replication. Now, I really don't known if latency monitor at DC3 could have an impact on osd read/write latency is this monitor elected ? vs stretched cluster with osd only use local dc monitors ? What is the advantage of stretch cluster here ? (with good redudant links between sites)

2 1

recovery_unfound during scrub with auto repair = true
by Dan van der Ster 13 Jun '21

13 Jun '21

Hi all, The cluster here is running v14.2.20 and is used for RBD images. We have a PG in recovery_unfound state and since this is the first time we've had this occur, we wanted to get your advice on the best course of action. PG 4.1904 went into state active+recovery_unfound+degraded+repair [1] during normal scrubbing (but note that we have `osd scrub auto repair = true`). 2021-06-13 03:15:11.559680 osd.951 (osd.951) 138 : cluster [DBG] 4.1904 repair starts 2021-06-13 04:00:49.369256 osd.951 (osd.951) 139 : cluster [ERR] 4.1904 shard 951 soid 4:209cfddb:::rbd_data.3a4ff12d847b61.000000000001c39e:head : candidate had a read error The scrub detected a read error on the primary of this PG, and tried to repair it by reading from the other 2 osds: Jun 13 04:00:46 xxx kernel: sd 0:0:25:0: [sdp] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DR Jun 13 04:00:46 xxx kernel: sd 0:0:25:0: [sdp] tag#6 Sense Key : Medium Error [current] [descript Jun 13 04:00:46 xxx kernel: sd 0:0:25:0: [sdp] tag#6 Add. Sense: Unrecovered read error Jun 13 04:00:46 xxx kernel: sd 0:0:25:0: [sdp] tag#6 CDB: Read(16) 88 00 00 00 00 02 ba 8c 0b 00 Jun 13 04:00:46 xxx kernel: blk_update_request: critical medium error, dev sdp, sector 1171967531 But it seems that the other 2 osds could not repair this failed read on the primary because they don't have the correct version of the object: 2021-06-13 04:28:29.412765 osd.951 (osd.951) 140 : cluster [ERR] 4.1904 repair 0 missing, 1 inconsistent objects 2021-06-13 04:28:29.413320 osd.951 (osd.951) 141 : cluster [ERR] 4.1904 repair 1 errors, 1 fixed 2021-06-13 04:28:29.445659 osd.14 (osd.14) 414 : cluster [ERR] 4.1904 push 4:209cfddb:::rbd_data.3a4ff12d847b61.000000000001c39e:head v 3592634'367863320 failed because local copy is 3593555'368312656 2021-06-13 04:28:29.472554 osd.344 (osd.344) 124 : cluster [ERR] 4.1904 push 4:209cfddb:::rbd_data.3a4ff12d847b61.000000000001c39e:head v 3592634'367863320 failed because local copy is 3593555'368312656 2021-06-13 04:28:30.863807 mgr.yyy (mgr.692832499) 648287 : cluster [DBG] pgmap v557097: 19456 pgs: 1 active+recovery_unfound+degraded+repair, 2 active+clean+scrubbing, 19423 active+clean, 30 active+clean+scrubbing+deep+repair; 1.3 PiB data, 4.0 PiB used, 2.1 PiB / 6.1 PiB avail; 350 MiB/s rd, 766 MiB/s wr, 16.93k op/s; 3/1063641423 objects degraded (0.000%); 1/354547141 objects unfound (0.000%) I don't understand how the versions of the objects would get out of sync -- there have been no other recent failures on these disks, AFAICT. So my best guess is that the IO error on 951 confused the repair process -- the osd.951 tried to recover the non-latest version of the object. (This would imply that the object versions on osds 14 and 344 are in fact the correct newest versions). We have a few ideas how to fix this: * osd 951 is sick, so drain it by setting `ceph osd primary-affinity 951 0` and `ceph osd out 951` * osd 951 is really sick, so just stop it now and backfill its PGs to other OSDs. * Don't stop osd 951 yet: Restart all three relevant OSDs and see if that fixes the object versions. * Don't drain osd 951 yet: Make OSD 14 or 344 the primary for this PG, (e.g. ceph osd primary-affinity 951 0) then run `ceph pg repair 4.1904` so that the version from osds 14/344 can be pushed. * Use mark_unfound_lost revert, or delete. (and inform the user their image to fsck their image). Does anyone have some recent experience or advice on this issue? Best Regards, Dan [1] # ceph pg 4.1904 query { "state": "active+recovery_unfound+degraded+repair", "snap_trimq": "[1c7fd~1,1c7ff~1,1c801~1,1c803~1,1c805~1]", "snap_trimq_len": 5, "epoch": 3593586, "up": [ 951, 344, 14 ], "acting": [ 951, 344, 14 ], "acting_recovery_backfill": [ "14", "344", "951" ], ...

1 2

bluestore label returned: (2) No such file or directory
by Karl Mardoff Kittilsen 13 Jun '21

13 Jun '21

Hi List I have a osd (83) that fails to start. It is made up of one 4TB drive and an 80GB DB on nvme. There was a cluster-full situation that is now solved, however I am quite sure the issue with this particular osd is unrelated. When I try to start the osd it failes to read the label of the block.db device failing with the followint lines: 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-83/block.db size 80 GiB -1 bluestore(/var/lib/ceph/osd/ceph-83) _minimal_open_bluefs check block device(/var/lib/ceph/osd/ceph-83/block.db) label returned: (2) No such file or directory 1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) close 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) close -1 osd.83 0 OSD:init: unable to mount object store -1 ** ERROR: osd init failed: (2) No such file or directory (Full log: https://gist.github.com/NightDog/7b50349da1410bb05bd7f4d54a02f055) The last thing that happened to the OSD before it started to fail booting was it being terminated during what I believe was a reboot: received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Last lines++ of the last successful run: https://gist.github.com/NightDog/fd9b4b7b3e0c0c2ba29ce5d325bb97c6 When I try to run ceph-bluestore-tool --log-level 30 show-label on the block.db it returns: "unable to read label for /dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be: (2) No such file or directory" The block returns the label fine (see master gist): https://gist.github.com/NightDog/4518bf11b364170911e5743b5ed0f614 The strange thing is however that lvs -o lv_tags returns just fine for the block.db: root@ceph-node201:~# lvs -o lv_tags /dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be LV Tags ceph.block_device=/dev/ceph-ff60b68a-26fe-4294-8bec-4a9c329e858d/osd-block-73ab12e6-7758-4ebe-9319-5935309fcacd,ceph.block_uuid=nbRXYl-fRrQ-qyYP-D93c-IGct-yKg4-rujDOX,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=f4495398-a8c4-4ad9-8219-80c48625abdf,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be,ceph.db_uuid=K09p3L-QV06-LOLO-uVeT-2ulz-GD3O-CEyRcs,ceph.encrypted=0,ceph.osd_fsid=73ab12e6-7758-4ebe-9319-5935309fcacd,ceph.osd_id=83,ceph.osdspec_affinity=osd-spec-2xx,ceph.type=db,ceph.vdo=0 So it seems to me that for some reason, ceph-bluestore-tool fails to read the label of the block.db device, even tho it is there, and then fails the startup of the OSD. Trying to write keys with ceph-bluestore-tool set-label-key fails with the same error message. I see no reason why there should be any damage to either the .db or block device, and since the labels are there in LVM, guess ceph-bluestore-tool errors out on something else? Would it be possible to get some help with regards to getting this .db and OSD back up again? Thanks! PS: Running version 15.2.8, also tried with 16.2.3-> show-label, with same result. -- Regards Karl M. Kittilsen

1 0

Ceph Poor RBD Performance
by Eren Cankurtaran 12 Jun '21

12 Jun '21

Hi. I'm having performance issue about ceph rbd. The performance is not i expected according to my node metrics. here're the metrics. I've used Calico as CNI. version: Rook-ceph 1.6 I've used stock yaml files and rook is not running on host network Centos 8 Stream [root@node4 ~]# uname -a Linux node4 4.18.0-240.el8.x86_64 #1 SMP Fri Sep 25 19:48:47 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@rook-ceph-tools-fc5f9586c-nb2h7 /]# ceph health HEALTH_WARN mons are allowing insecure global_id reclaim; clock skew detected on mon.b, mon.c; 1 pool(s) do not have an application enabled [root@node1 ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:40:09Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:32:49Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"} [root@node4 ~]# iperf3 -c 172.16.11.181 -p 3000 M Connecting to host 172.16.11.181, port 3000 [ 5] local 172.16.11.180 port 46390 connected to 172.16.11.181 port 3000 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 201 MBytes 1.69 Gbits/sec 488 48.1 KBytes [ 5] 1.00-2.00 sec 160 MBytes 1.34 Gbits/sec 397 39.6 KBytes [ 5] 2.00-3.00 sec 201 MBytes 1.68 Gbits/sec 513 69.3 KBytes [ 5] 3.00-4.00 sec 200 MBytes 1.68 Gbits/sec 374 38.2 KBytes [ 5] 4.00-5.00 sec 199 MBytes 1.67 Gbits/sec 402 48.1 KBytes [ 5] 5.00-6.00 sec 201 MBytes 1.69 Gbits/sec 559 48.1 KBytes [ 5] 6.00-7.00 sec 204 MBytes 1.71 Gbits/sec 470 45.2 KBytes [ 5] 7.00-8.00 sec 199 MBytes 1.67 Gbits/sec 575 46.7 KBytes [ 5] 8.00-9.00 sec 200 MBytes 1.68 Gbits/sec 404 49.5 KBytes [ 5] 9.00-10.00 sec 200 MBytes 1.68 Gbits/sec 391 49.5 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.92 GBytes 1.65 Gbits/sec 4573 sender [ 5] 0.00-10.04 sec 1.92 GBytes 1.64 Gbits/sec receiver *In pod that mounted rbd volume cassandra@k8ssandra-dc1-default-sts-0:/var/lib/cassandra$ dd if=/dev/zero of=/var/lib/cassandra/test.img bs=1G count=1 oflag=dsync 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 28.051 s, 38.3 MB/s [root@rook-ceph-tools-fc5f9586c-nb2h7 /]# ceph status cluster: id: d6584e08-f8f5-43e4-a258-8d652cc28e0a health: HEALTH_WARN mons are allowing insecure global_id reclaim clock skew detected on mon.b, mon.c 1 pool(s) do not have an application enabled services: mon: 3 daemons, quorum a,b,c (age 32h) mgr: a(active, since 3h) osd: 24 osds: 24 up (since 32h), 24 in (since 3d) data: pools: 4 pools, 97 pgs objects: 181.38k objects, 686 GiB usage: 1.3 TiB used, 173 TiB / 175 TiB avail pgs: 97 active+clean io: client: 308 KiB/s wr, 0 op/s rd, 4 op/s wr [root@rook-ceph-tools-fc5f9586c-nb2h7 /]# rados bench -p scbench 15 write seq --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 15 seconds or 0 objects Object prefix: benchmark_data_rook-ceph-tools-fc5f9586c-nb2_37098 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 19 3 11.9985 12 0.480534 0.377567 2 16 22 6 11.9986 12 1.77015 1.04767 3 16 27 11 14.6652 20 2.48028 1.26595 4 16 29 13 12.9988 8 0.766731 1.18763 5 16 34 18 14.3986 20 4.95184 1.44111 6 16 41 25 16.6651 28 0.742371 1.91373 7 16 45 29 16.5699 16 1.00815 2.14879 8 16 49 33 16.4985 16 5.65826 2.1447 9 16 57 41 18.2206 32 0.941691 2.46741 10 16 65 49 19.5982 32 4.63243 2.45243 11 16 70 54 19.6346 20 0.122698 2.4876 12 16 74 58 19.3316 16 0.700267 2.50753 13 16 76 60 18.4599 8 1.3582 2.54338 14 16 78 62 17.7127 8 5.65752 2.63185 15 16 80 64 17.0651 8 5.25659 2.77212 16 15 80 65 16.2485 4 5.95131 2.82103 17 11 80 69 16.2337 16 10.4721 3.09282 18 6 80 74 16.4429 20 7.71085 3.23797 19 1 80 79 16.63 20 5.98475 3.47738 Total time run: 19.0674 Total writes made: 80 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 16.7826 Stddev Bandwidth: 8.02919 Max bandwidth (MB/sec): 32 Min bandwidth (MB/sec): 4 Average IOPS: 4 Stddev IOPS: 2.0073 Max IOPS: 8 Min IOPS: 1 Average Latency(s): 3.51805 Stddev Latency(s): 2.97502 Max latency(s): 10.4721 Min latency(s): 0.104524 [root@rook-ceph-tools-fc5f9586c-nb2h7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 174.65735 root default -5 36.38695 host node4 0 hdd 7.27739 osd.0 up 1.00000 1.00000 4 hdd 7.27739 osd.4 up 1.00000 1.00000 8 hdd 7.27739 osd.8 up 1.00000 1.00000 12 hdd 7.27739 osd.12 up 1.00000 1.00000 16 hdd 7.27739 osd.16 up 1.00000 1.00000 -9 36.38695 host node5 2 hdd 7.27739 osd.2 up 1.00000 1.00000 6 hdd 7.27739 osd.6 up 1.00000 1.00000 10 hdd 7.27739 osd.10 up 1.00000 1.00000 14 hdd 7.27739 osd.14 up 1.00000 1.00000 18 hdd 7.27739 osd.18 up 1.00000 1.00000 -7 36.38695 host node6 1 hdd 7.27739 osd.1 up 1.00000 1.00000 5 hdd 7.27739 osd.5 up 1.00000 1.00000 9 hdd 7.27739 osd.9 up 1.00000 1.00000 13 hdd 7.27739 osd.13 up 1.00000 1.00000 17 hdd 7.27739 osd.17 up 1.00000 1.00000 -3 29.10956 host node7 3 hdd 7.27739 osd.3 up 1.00000 1.00000 7 hdd 7.27739 osd.7 up 1.00000 1.00000 11 hdd 7.27739 osd.11 up 1.00000 1.00000 15 hdd 7.27739 osd.15 up 1.00000 1.00000 -11 36.38695 host node8 19 hdd 7.27739 osd.19 up 1.00000 1.00000 20 hdd 7.27739 osd.20 up 1.00000 1.00000 21 hdd 7.27739 osd.21 up 1.00000 1.00000 22 hdd 7.27739 osd.22 up 1.00000 1.00000 23 hdd 7.27739 osd.23 up 1.00000 1.00000 [root@node1 ~]# kubectl get pods -n rook-ceph NAME READY STATUS RESTARTS AGE csi-cephfsplugin-9x4dc 3/3 Running 9 14d csi-cephfsplugin-cv7k9 3/3 Running 9 14d csi-cephfsplugin-f6s4k 3/3 Running 22 14d csi-cephfsplugin-fn9w4 3/3 Running 12 14d csi-cephfsplugin-provisioner-59499cbcdd-txtjq 6/6 Running 0 33h csi-cephfsplugin-provisioner-59499cbcdd-wj9dx 6/6 Running 11 4d7h csi-cephfsplugin-tdp99 3/3 Running 9 14d csi-rbdplugin-4bv84 3/3 Running 12 14d csi-rbdplugin-ddgq8 3/3 Running 22 14d csi-rbdplugin-hnnx2 3/3 Running 10 14d csi-rbdplugin-provisioner-857d65496c-qzbcq 6/6 Running 0 33h csi-rbdplugin-provisioner-857d65496c-s6nnd 6/6 Running 9 4d7h csi-rbdplugin-prthz 3/3 Running 9 14d csi-rbdplugin-tjj7z 3/3 Running 12 14d rook-ceph-crashcollector-node4-5bb88d4866-rfv54 1/1 Running 1 4d7h rook-ceph-crashcollector-node5-75dcbb9f45-5tlv8 1/1 Running 0 33h rook-ceph-crashcollector-node6-777c9f579b-zzf69 1/1 Running 0 33h rook-ceph-crashcollector-node7-84464d7cc5-2zthq 1/1 Running 0 33h rook-ceph-crashcollector-node8-66bcdc9c4b-drcj2 1/1 Running 0 3d12h rook-ceph-mgr-a-fb794cf97-cbmws 1/1 Running 26 34h rook-ceph-mon-a-58bf5db667-lhv26 1/1 Running 0 33h rook-ceph-mon-b-78df76b9f8-p7w69 1/1 Running 0 33h rook-ceph-mon-c-6fc84459fd-nc8hx 1/1 Running 0 4d8h rook-ceph-operator-65965c66b5-z5xfl 1/1 Running 2 4d7h rook-ceph-osd-0-59747d74d6-lg5lh 1/1 Running 16 4d7h rook-ceph-osd-1-56c66f7d49-vxj9m 1/1 Running 0 33h rook-ceph-osd-10-6cf5f76d57-g5jhb 1/1 Running 0 33h rook-ceph-osd-11-5446ff487f-qsdmw 1/1 Running 0 33h rook-ceph-osd-12-db4859779-5vrvm 1/1 Running 8 4d7h rook-ceph-osd-13-65d4859986-zx8kx 1/1 Running 0 33h rook-ceph-osd-14-5c6db6c8c9-ghhtk 1/1 Running 0 33h rook-ceph-osd-15-568c6bbf64-m2f2v 1/1 Running 0 33h rook-ceph-osd-16-7d475cc4cc-9lzkp 1/1 Running 8 4d7h rook-ceph-osd-17-6487d4ff79-5x8bh 1/1 Running 0 33h rook-ceph-osd-18-7dd896c985-bjvbz 1/1 Running 0 33h rook-ceph-osd-19-5ccfbf896f-gll6k 1/1 Running 1 4d8h rook-ceph-osd-2-7db5897c7b-fxc4k 1/1 Running 0 33h rook-ceph-osd-20-7b7475f95c-cfprr 1/1 Running 1 4d8h rook-ceph-osd-21-7c544458cf-mfbzr 1/1 Running 0 4d8h rook-ceph-osd-22-76ff6df84-tf4hx 1/1 Running 0 4d8h rook-ceph-osd-23-86b976fdb4-c5xz9 1/1 Running 0 4d8h rook-ceph-osd-3-6ccbd79b9d-dvrwj 1/1 Running 0 33h rook-ceph-osd-4-56c4c85686-pm488 1/1 Running 9 4d7h rook-ceph-osd-5-75dd798967-rmhrp 1/1 Running 0 33h rook-ceph-osd-6-f66fd44fb-f2h2d 1/1 Running 0 33h rook-ceph-osd-7-6fbb57486c-4xvzh 1/1 Running 0 33h rook-ceph-osd-8-59567ccf6-xnmx7 1/1 Running 8 4d7h rook-ceph-osd-9-5b9bc76c98-f8dbf 1/1 Running 0 33h rook-ceph-osd-prepare-node4-jgwvb 0/1 Completed 0 3h1m rook-ceph-osd-prepare-node5-jwvsk 0/1 Completed 0 3h1m rook-ceph-osd-prepare-node6-r2ps9 0/1 Completed 0 3h1m rook-ceph-osd-prepare-node7-cmfsf 0/1 Completed 0 3h1m rook-ceph-osd-prepare-node8-kk256 0/1 Completed 0 3h1m rook-ceph-tools-fc5f9586c-nb2h7 1/1 Running 0 34h

1 0

Kubernetes - How to create a PersistentVolume on an existing durable ceph volume?
by Ralph Soika 12 Jun '21

12 Jun '21

Hi, I have setup a ceph cluster (octopus) and installed he rbd plugins/provisioner in my Kubernetes cluster. I can create dynamically FS and Block Volumes which is fine. For that I have created the following the following storageClass apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph provisioner: rbd.csi.ceph.com parameters: clusterID: <clusterID> pool: kubernetes imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: ceph-system csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: ceph-system csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: ceph-system reclaimPolicy: Delete allowVolumeExpansion: true mountOptions: - discard This works fine for ephemeral dynamically crated volumes. But now I want to use durable volume with the reclaimPolicy:Retain. I expect that I need to create the image in my kubernetes pool on the ceph cluster first - which I have done. I defined the following new storage class with the reclaimPolicy 'Retain': apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-durable provisioner: rbd.csi.ceph.com parameters: clusterID: <clusterID> pool: kubernetes imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: ceph-system csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: ceph-system csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: ceph-system reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - discard And finally I created the following PersistentVolume and PersistentVolumeClaim: --- kind: PersistentVolume apiVersion: v1 metadata: name: demo-internal-index spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: - ReadWriteOnce claimRef: namespace: office-demo-internal name: index csi: driver: driver.ceph.io fsType: ext4 volumeHandle: demo-internal-index storageClassName: ceph-durable --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: index namespace: office-demo-internal spec: accessModes: - ReadWriteOnce storageClassName: ceph-durable resources: requests: storage: 1Gi volumeName: "demo-internal-index" But this seems not to work and I can see the following deployment warning: attachdetach-controller AttachVolume.Attach failed for volume "demo-internal-index" : attachdetachment timeout for volume demo-internal-index But the PV exists: $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE demo-internal-index 1Gi RWO Retain Bound office-demo-internal/index ceph-durable 2m35s and also the PVC exists: $ kubectl get pvc -n office-demo-internal NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE index Bound demo-internal-index 1Gi RWO ceph-durable 53m I guess my PV object is nonsense? Can someone provide me an example how to setup the PV object in Kubernetes. I only found examples where the Ceph Monitoring IPs and the user/password is configured within the PV object. But I would expect that this is covered by the storage class already? Thanks for your help === Ralph

1 1

Error on Ceph Dashboard
by Robert W. Eckert 11 Jun '21

11 Jun '21

Hi - this just started happening in the past few days using Ceph Pacific 16.2.4 via cephadmin (Podman containers) The dashboard is returning No active ceph-mgr instance is currently running the dashboard. A failover may be in progress. Retrying in 5 seconds... And ceph status returns cluster: id: fe3a7cb0-69ca-11eb-8d45-c86000d08867 health: HEALTH_WARN Module 'dashboard' has failed dependency: cannot import name 'AuthManager' clock skew detected on mon.cube services: mon: 3 daemons, quorum story,cube,rhel1 (age 46h) mgr: cube.tvlgnp(active, since 47h), standbys: rhel1.zpzsjc, story.gffann mds: 2/2 daemons up, 1 standby osd: 13 osds: 13 up (since 46h), 13 in (since 46h) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 497 pgs objects: 1.50M objects, 2.1 TiB usage: 6.2 TiB used, 32 TiB / 38 TiB avail pgs: 497 active+clean io: client: 255 B/s rd, 2.7 KiB/s wr, 0 op/s rd, 0 op/s wr The only thing that has happened on the cluster was one of the servers was rebooted. No configuration changes were performed Any suggestions? Thanks, rob

2 3

driver name rbd.csi.ceph.com not found in the list of registered CSI drivers ?
by Ralph Soika 11 Jun '21

11 Jun '21

Hi, I try to connect my new ceph cluster (octopus) with my kubernetes system. Therefor I followed the setup guide form the official documentation: https://docs.ceph.com/en/octopus/rbd/rbd-kubernetes/ The csi-rbdplugin-provisioner is running successful on all my kubernetes worker nodes (as far as I can see). Now I try to deploy the nginx example : --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ceph-pvc spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 1Gi storageClassName: ceph --- apiVersion: v1 kind: Pod metadata: name: csi-rbd-demo-pod spec: containers: - name: web-server image: nginx volumeMounts: - name: mypvc mountPath: /var/lib/www/html volumes: - name: mypvc persistentVolumeClaim: claimName: ceph-pvc readOnly: false A Persitence volume is created $ kubectl get pv .... pvc-5c64ed45-adde-4fe7-9b38-9d4c7a8f7d34 1Gi RWO Delete Bound default/ceph-pvc ceph 7m15s and also the persitence volume claim: $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-pvc Bound pvc-5c64ed45-adde-4fe7-9b38-9d4c7a8f7d34 1Gi RWO ceph 8m14s But the POD is not deployed, because of the following error message: MountVolume.MountDevice failed for volume "pvc-5c64ed45-adde-4fe7-9b38-9d4c7a8f7d34" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rbd.csi.ceph.com not found in the list of registered CSI drivers Can someone help me to understand the meaning of this error message? Did I need to install something else ? Maybe a ceph-fs-plugin...? Thanks for any help === Ralph --

1 1

suggestion for Ceph client network config
by Götz Reinicke 11 Jun '21

11 Jun '21

Hi all We get a new samba smb fileserver who mounts our cephfs for exporting some shares. What might be a good or better network setup for that server? Should I configure two interfaces - one for the smb share export towards our workstations and desktops and one towards the ceph cluster? Or would it be „ok“ for all traffic to be on one interface? The server has 40G ports. Thanks for your suggestions and feedback . Regards . Götz

2 1

lib remoto in ubuntu
by Alfredo Rezinovsky 11 Jun '21

11 Jun '21

I cannot enable cephadm because it cannot find remoto lib. Even when I installed it using "pip3 install remoto" and then installed ir from the deb package build from the git sources at https://github.com/alfredodeza/remoto/ If I type "import remoto" in a python3 prompt it works. -- Alfrenovsky

2 1

Ceph Ansible fails on check if monitor initial keyring already exists
by Jared Jacob 11 Jun '21

11 Jun '21

I am running the Ceph ansible script to install ceph version Stable-6.0 (Pacific). When running the sample yml file that was supplied by the github repo it runs fine up until the "ceph-mon : check if monitor initial keyring already exists" step. There it will hang for 30-40 minutes before failing. From my understanding ceph ansible should be creating this keyring and using it for communication between monitors, so does anyone know why the playbook would have a hard time with this step? Thanks in advance!

2 1