June 2021 - ceph-users - lists.ceph.io

Upgrading ceph to latest version, skipping minor versions?
by Matt Larson 15 Jun '21

15 Jun '21

Looking at the documentation for ( https://docs.ceph.com/en/latest/cephadm/upgrade/) - I have a question on whether you need to sequentially upgrade for each minor versions, 15.2.1 -> 15.2.3 -> ... -> 15.2.XX? Can you safely upgrade by directly specifying the latest version from several minor versions behind? ceph orch upgrade start --ceph-version 15.2.13 -Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

2 1

Updated ceph-osd package, now get -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
by Matt Larson 14 Jun '21

14 Jun '21

I was working through updates of CentOS and packages on each of the Ceph storage nodes of a cluster, and I hit an issue after I updated a package `ceph-osd` on the original Ceph node of the cluster. 1. Before the updates, the Ceph cluster was running 15.2.3 version 2. After the update of the package with `dnf install ceph-osd`, I was unable to check the status any more with `ceph -s`: 2021-06-14T14:01:53.216-0500 7efdef7fe700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] [errno 13] RADOS permission denied (error connecting to the cluster) 3. The `ceph -v` command now shows as version 15.2.13. Based on other reported issues this could be due recent changes with security fixes ( https://docs.ceph.com/en/latest/security/CVE-2021-20288/ ) that were introduced in 15.2.11. This cluster had originally been built with `cephadm` tool and containerized daemons. How can I restore the ability to connect with the command-line `ceph` client to check the status and all other interactions? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

1 1

Cephfs mount not recovering after icmp-not-reachable
by Simon Sutter 14 Jun '21

14 Jun '21

Hello everyone! We had a switch outage and the ceph kernel mount did not work anymore. This is the fstab entry: 10.99.10.1:/somefolder /cephfs ceph _netdev,nofail,name=cephcluster,secret=IsSecret 0 0 I reproduced it with disabling the vlan on the switch on which the ceph is reachable, which gives a icmp-not-reachable. I did this for five minutes, after that, "ls /cephfs" just gives a "permission denied" in dmesg i can see this: [ 1412.994921] libceph: mon1 10.99.10.4:6789 session lost, hunting for new mon [ 1413.009325] libceph: mon0 10.99.10.1:6789 session established [ 1452.998646] libceph: mon2 10.99.15.3:6789 session lost, hunting for new mon [ 1452.998679] libceph: mon0 10.99.10.1:6789 session lost, hunting for new mon [ 1461.989549] libceph: mon4 10.99.15.5:6789 socket closed (con state CONNECTING) --- [ 1787.045148] libceph: mon3 10.99.15.4:6789 socket closed (con state CONNECTING) [ 1787.062587] libceph: mon0 10.99.10.1:6789 session established [ 1787.086103] libceph: mon4 10.99.15.5:6789 session established [ 1814.028761] libceph: mds0 10.99.10.4:6801 socket closed (con state OPEN) [ 1815.029811] libceph: mds0 10.99.10.4:6801 connection reset [ 1815.029829] libceph: reset on mds0 [ 1815.029831] ceph: mds0 closed our session [ 1815.029833] ceph: mds0 reconnect start [ 1815.052219] ceph: mds0 reconnect denied [ 1815.052229] ceph: dropping dirty Fw state for ffff9d9085da1340 1099512175611 [ 1815.052231] ceph: dropping dirty+flushing Fw state for ffff9d9085da1340 1099512175611 [ 1815.273008] libceph: mds0 10.99.10.4:6801 socket closed (con state NEGOTIATING) [ 1816.033241] ceph: mds0 rejected session [ 1829.018643] ceph: mds0 hung [ 1880.088504] ceph: mds0 came back [ 1880.088662] ceph: mds0 caps renewed [ 1880.094018] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null i_snap_realm [ 1881.100367] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null i_snap_realm [ 2046.768969] conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module. [ 2061.731126] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null i_snap_realm Is this a bug to report or wrong configuration? Did someone else had this before? To solve the problem, a simple remount does the trick. Thanks in advance Simon

3 2

CephFS design
by Szabo, Istvan (Agoda) 14 Jun '21

14 Jun '21

Hi, Can you suggest me what is a good cephfs design? I've never used it, only rgw and rbd we have, but want to give a try. Howvere in the mail list I saw a huge amount of issues with cephfs so would like to go with some let's say bulletproof best practices. Like separate the mds from mon and mgr? Need a lot of memory? Should be on ssd or nvme? How many cpu/disk ... Very appreciate it. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

7 11

cephadm failed in Pacific release: Unable to set up "admin" label
by Ralph Soika 14 Jun '21

14 Jun '21

hi, if you follow the latest install guide here <https://docs.ceph.com/en/latest/cephadm/install/> to install pacific release, the bootstrap command will print the following error message: .... ...... mgr epoch 13 is available Generating a dashboard self-signed certificate... Creating initial admin user... Fetching dashboard port number... Ceph Dashboard is now available at: URL: https://ceph-1:8443/ User: admin Password: xxx Enabling client.admin keyring and conf on hosts with "admin" label Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16 -e NODE_NAME=tikal-ceph-1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d6c1ba28-cd0d-11eb-8b39-960000bd038e:/var/log/ceph:z -v /tmp/ceph-tmp9t3y8y39:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaepq1rte:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16 orch client-keyring set client.admin label:_admin /usr/bin/ceph: stderr Invalid command: client-keyring not in start|stop|restart|redeploy|reconfig /usr/bin/ceph: stderr orch start|stop|restart|redeploy|reconfig <service_name> : Start, stop, restart, redeploy, or reconfig an entire service (i.e. all daemons) /usr/bin/ceph: stderr Error EINVAL: invalid command Unable to set up "admin" label; assuming older version of Ceph ...... .... As a result the first node is not working correctly and you can not add mom hosts. I tried to set the label manually afterwards with : # ceph orch host label add ceph-1 _admin But this seems to have no effect. Is this problem known? So only solution seems to stay with the octopus release :-/ === Ralph

2 3

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Creating a role in another tenant seems to be possible
by Daniel Iwan 14 Jun '21

14 Jun '21

Hi It seems that with command like this aws --profile=my-user-tenant1 --endpoint=$HOST_S3_API --region="" iam create-role --role-name="tenant2\$TemporaryRole" --assume-role-policy-document file://json/trust-policy-assume-role.json I can create a role in another tenant. Executing user have roles:* capability which I think is necessary to be able to create roles, but at the same time it seems to be a global ability, for all tenants. Similarly, a federated user who assumes a role with iam:CreateRole permission can create an arbitrary role like below. aws --endpoint=$HOST_S3_API --region="" iam create-role --role-name="tenant2\$TemporaryRole" --assume-role-policy-document file://json/trust-policy-assume-role.json Example permission policy { "Statement":[ {"Effect":"Allow","Action":["iam:GetRole"]}, {"Effect":"Allow","Action":["iam:CreateRole"]} ] } Capability roles:* is not needed in this case, which I think is correct, because only permission policy of the assumed role is checked. Getting information about a role from other tenants is possible with iam:GetRole. This is less controversial but I would still expect it to be scoped to the user's tenant unless explicit tenant name is stated in the policy like this {"Effect":"Allow","Action":["iam:GetRole"],"Resource":"arn:aws:iam::tenant2:*"} Possibly I'm missing something. Why is crossing tenants possible? Regards Daniel

2 4

stretched cluster or not, with mon in 3 DC and osds on 2 DC
by aderumier＠odiso.com 14 Jun '21

14 Jun '21

Hi, I'm currently reading the documentation about stretched cluster, I would like to known if it's needed or not with this kind of 3 dc setup: 3km (0.2ms) DC1--------------DC2 30km(3ms) | | 30km (2-3ms) |--------DC3------ DC1 && DC2 are near each other, small latency. (0.2ms) DC3 is at 30km with bigger latency. (2ms) separated links between dc with different physical path 1 monitor on each dc osd on DC1/DC2, with size=4 Cluster is full nvme or ssd, lowest latency is required for osd replication. Now, I really don't known if latency monitor at DC3 could have an impact on osd read/write latency is this monitor elected ? vs stretched cluster with osd only use local dc monitors ? What is the advantage of stretch cluster here ? (with good redudant links between sites)

2 1

recovery_unfound during scrub with auto repair = true
by Dan van der Ster 13 Jun '21

13 Jun '21

Hi all, The cluster here is running v14.2.20 and is used for RBD images. We have a PG in recovery_unfound state and since this is the first time we've had this occur, we wanted to get your advice on the best course of action. PG 4.1904 went into state active+recovery_unfound+degraded+repair [1] during normal scrubbing (but note that we have `osd scrub auto repair = true`). 2021-06-13 03:15:11.559680 osd.951 (osd.951) 138 : cluster [DBG] 4.1904 repair starts 2021-06-13 04:00:49.369256 osd.951 (osd.951) 139 : cluster [ERR] 4.1904 shard 951 soid 4:209cfddb:::rbd_data.3a4ff12d847b61.000000000001c39e:head : candidate had a read error The scrub detected a read error on the primary of this PG, and tried to repair it by reading from the other 2 osds: Jun 13 04:00:46 xxx kernel: sd 0:0:25:0: [sdp] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DR Jun 13 04:00:46 xxx kernel: sd 0:0:25:0: [sdp] tag#6 Sense Key : Medium Error [current] [descript Jun 13 04:00:46 xxx kernel: sd 0:0:25:0: [sdp] tag#6 Add. Sense: Unrecovered read error Jun 13 04:00:46 xxx kernel: sd 0:0:25:0: [sdp] tag#6 CDB: Read(16) 88 00 00 00 00 02 ba 8c 0b 00 Jun 13 04:00:46 xxx kernel: blk_update_request: critical medium error, dev sdp, sector 1171967531 But it seems that the other 2 osds could not repair this failed read on the primary because they don't have the correct version of the object: 2021-06-13 04:28:29.412765 osd.951 (osd.951) 140 : cluster [ERR] 4.1904 repair 0 missing, 1 inconsistent objects 2021-06-13 04:28:29.413320 osd.951 (osd.951) 141 : cluster [ERR] 4.1904 repair 1 errors, 1 fixed 2021-06-13 04:28:29.445659 osd.14 (osd.14) 414 : cluster [ERR] 4.1904 push 4:209cfddb:::rbd_data.3a4ff12d847b61.000000000001c39e:head v 3592634'367863320 failed because local copy is 3593555'368312656 2021-06-13 04:28:29.472554 osd.344 (osd.344) 124 : cluster [ERR] 4.1904 push 4:209cfddb:::rbd_data.3a4ff12d847b61.000000000001c39e:head v 3592634'367863320 failed because local copy is 3593555'368312656 2021-06-13 04:28:30.863807 mgr.yyy (mgr.692832499) 648287 : cluster [DBG] pgmap v557097: 19456 pgs: 1 active+recovery_unfound+degraded+repair, 2 active+clean+scrubbing, 19423 active+clean, 30 active+clean+scrubbing+deep+repair; 1.3 PiB data, 4.0 PiB used, 2.1 PiB / 6.1 PiB avail; 350 MiB/s rd, 766 MiB/s wr, 16.93k op/s; 3/1063641423 objects degraded (0.000%); 1/354547141 objects unfound (0.000%) I don't understand how the versions of the objects would get out of sync -- there have been no other recent failures on these disks, AFAICT. So my best guess is that the IO error on 951 confused the repair process -- the osd.951 tried to recover the non-latest version of the object. (This would imply that the object versions on osds 14 and 344 are in fact the correct newest versions). We have a few ideas how to fix this: * osd 951 is sick, so drain it by setting `ceph osd primary-affinity 951 0` and `ceph osd out 951` * osd 951 is really sick, so just stop it now and backfill its PGs to other OSDs. * Don't stop osd 951 yet: Restart all three relevant OSDs and see if that fixes the object versions. * Don't drain osd 951 yet: Make OSD 14 or 344 the primary for this PG, (e.g. ceph osd primary-affinity 951 0) then run `ceph pg repair 4.1904` so that the version from osds 14/344 can be pushed. * Use mark_unfound_lost revert, or delete. (and inform the user their image to fsck their image). Does anyone have some recent experience or advice on this issue? Best Regards, Dan [1] # ceph pg 4.1904 query { "state": "active+recovery_unfound+degraded+repair", "snap_trimq": "[1c7fd~1,1c7ff~1,1c801~1,1c803~1,1c805~1]", "snap_trimq_len": 5, "epoch": 3593586, "up": [ 951, 344, 14 ], "acting": [ 951, 344, 14 ], "acting_recovery_backfill": [ "14", "344", "951" ], ...

1 2

bluestore label returned: (2) No such file or directory
by Karl Mardoff Kittilsen 13 Jun '21

13 Jun '21

Hi List I have a osd (83) that fails to start. It is made up of one 4TB drive and an 80GB DB on nvme. There was a cluster-full situation that is now solved, however I am quite sure the issue with this particular osd is unrelated. When I try to start the osd it failes to read the label of the block.db device failing with the followint lines: 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-83/block.db size 80 GiB -1 bluestore(/var/lib/ceph/osd/ceph-83) _minimal_open_bluefs check block device(/var/lib/ceph/osd/ceph-83/block.db) label returned: (2) No such file or directory 1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) close 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) close -1 osd.83 0 OSD:init: unable to mount object store -1 ** ERROR: osd init failed: (2) No such file or directory (Full log: https://gist.github.com/NightDog/7b50349da1410bb05bd7f4d54a02f055) The last thing that happened to the OSD before it started to fail booting was it being terminated during what I believe was a reboot: received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Last lines++ of the last successful run: https://gist.github.com/NightDog/fd9b4b7b3e0c0c2ba29ce5d325bb97c6 When I try to run ceph-bluestore-tool --log-level 30 show-label on the block.db it returns: "unable to read label for /dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be: (2) No such file or directory" The block returns the label fine (see master gist): https://gist.github.com/NightDog/4518bf11b364170911e5743b5ed0f614 The strange thing is however that lvs -o lv_tags returns just fine for the block.db: root@ceph-node201:~# lvs -o lv_tags /dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be LV Tags ceph.block_device=/dev/ceph-ff60b68a-26fe-4294-8bec-4a9c329e858d/osd-block-73ab12e6-7758-4ebe-9319-5935309fcacd,ceph.block_uuid=nbRXYl-fRrQ-qyYP-D93c-IGct-yKg4-rujDOX,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=f4495398-a8c4-4ad9-8219-80c48625abdf,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be,ceph.db_uuid=K09p3L-QV06-LOLO-uVeT-2ulz-GD3O-CEyRcs,ceph.encrypted=0,ceph.osd_fsid=73ab12e6-7758-4ebe-9319-5935309fcacd,ceph.osd_id=83,ceph.osdspec_affinity=osd-spec-2xx,ceph.type=db,ceph.vdo=0 So it seems to me that for some reason, ceph-bluestore-tool fails to read the label of the block.db device, even tho it is there, and then fails the startup of the OSD. Trying to write keys with ceph-bluestore-tool set-label-key fails with the same error message. I see no reason why there should be any damage to either the .db or block device, and since the labels are there in LVM, guess ceph-bluestore-tool errors out on something else? Would it be possible to get some help with regards to getting this .db and OSD back up again? Thanks! PS: Running version 15.2.8, also tried with 16.2.3-> show-label, with same result. -- Regards Karl M. Kittilsen

1 0

Ceph Poor RBD Performance
by Eren Cankurtaran 12 Jun '21

12 Jun '21

Hi. I'm having performance issue about ceph rbd. The performance is not i expected according to my node metrics. here're the metrics. I've used Calico as CNI. version: Rook-ceph 1.6 I've used stock yaml files and rook is not running on host network Centos 8 Stream [root@node4 ~]# uname -a Linux node4 4.18.0-240.el8.x86_64 #1 SMP Fri Sep 25 19:48:47 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@rook-ceph-tools-fc5f9586c-nb2h7 /]# ceph health HEALTH_WARN mons are allowing insecure global_id reclaim; clock skew detected on mon.b, mon.c; 1 pool(s) do not have an application enabled [root@node1 ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:40:09Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:32:49Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"} [root@node4 ~]# iperf3 -c 172.16.11.181 -p 3000 M Connecting to host 172.16.11.181, port 3000 [ 5] local 172.16.11.180 port 46390 connected to 172.16.11.181 port 3000 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 201 MBytes 1.69 Gbits/sec 488 48.1 KBytes [ 5] 1.00-2.00 sec 160 MBytes 1.34 Gbits/sec 397 39.6 KBytes [ 5] 2.00-3.00 sec 201 MBytes 1.68 Gbits/sec 513 69.3 KBytes [ 5] 3.00-4.00 sec 200 MBytes 1.68 Gbits/sec 374 38.2 KBytes [ 5] 4.00-5.00 sec 199 MBytes 1.67 Gbits/sec 402 48.1 KBytes [ 5] 5.00-6.00 sec 201 MBytes 1.69 Gbits/sec 559 48.1 KBytes [ 5] 6.00-7.00 sec 204 MBytes 1.71 Gbits/sec 470 45.2 KBytes [ 5] 7.00-8.00 sec 199 MBytes 1.67 Gbits/sec 575 46.7 KBytes [ 5] 8.00-9.00 sec 200 MBytes 1.68 Gbits/sec 404 49.5 KBytes [ 5] 9.00-10.00 sec 200 MBytes 1.68 Gbits/sec 391 49.5 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.92 GBytes 1.65 Gbits/sec 4573 sender [ 5] 0.00-10.04 sec 1.92 GBytes 1.64 Gbits/sec receiver *In pod that mounted rbd volume cassandra@k8ssandra-dc1-default-sts-0:/var/lib/cassandra$ dd if=/dev/zero of=/var/lib/cassandra/test.img bs=1G count=1 oflag=dsync 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 28.051 s, 38.3 MB/s [root@rook-ceph-tools-fc5f9586c-nb2h7 /]# ceph status cluster: id: d6584e08-f8f5-43e4-a258-8d652cc28e0a health: HEALTH_WARN mons are allowing insecure global_id reclaim clock skew detected on mon.b, mon.c 1 pool(s) do not have an application enabled services: mon: 3 daemons, quorum a,b,c (age 32h) mgr: a(active, since 3h) osd: 24 osds: 24 up (since 32h), 24 in (since 3d) data: pools: 4 pools, 97 pgs objects: 181.38k objects, 686 GiB usage: 1.3 TiB used, 173 TiB / 175 TiB avail pgs: 97 active+clean io: client: 308 KiB/s wr, 0 op/s rd, 4 op/s wr [root@rook-ceph-tools-fc5f9586c-nb2h7 /]# rados bench -p scbench 15 write seq --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 15 seconds or 0 objects Object prefix: benchmark_data_rook-ceph-tools-fc5f9586c-nb2_37098 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 19 3 11.9985 12 0.480534 0.377567 2 16 22 6 11.9986 12 1.77015 1.04767 3 16 27 11 14.6652 20 2.48028 1.26595 4 16 29 13 12.9988 8 0.766731 1.18763 5 16 34 18 14.3986 20 4.95184 1.44111 6 16 41 25 16.6651 28 0.742371 1.91373 7 16 45 29 16.5699 16 1.00815 2.14879 8 16 49 33 16.4985 16 5.65826 2.1447 9 16 57 41 18.2206 32 0.941691 2.46741 10 16 65 49 19.5982 32 4.63243 2.45243 11 16 70 54 19.6346 20 0.122698 2.4876 12 16 74 58 19.3316 16 0.700267 2.50753 13 16 76 60 18.4599 8 1.3582 2.54338 14 16 78 62 17.7127 8 5.65752 2.63185 15 16 80 64 17.0651 8 5.25659 2.77212 16 15 80 65 16.2485 4 5.95131 2.82103 17 11 80 69 16.2337 16 10.4721 3.09282 18 6 80 74 16.4429 20 7.71085 3.23797 19 1 80 79 16.63 20 5.98475 3.47738 Total time run: 19.0674 Total writes made: 80 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 16.7826 Stddev Bandwidth: 8.02919 Max bandwidth (MB/sec): 32 Min bandwidth (MB/sec): 4 Average IOPS: 4 Stddev IOPS: 2.0073 Max IOPS: 8 Min IOPS: 1 Average Latency(s): 3.51805 Stddev Latency(s): 2.97502 Max latency(s): 10.4721 Min latency(s): 0.104524 [root@rook-ceph-tools-fc5f9586c-nb2h7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 174.65735 root default -5 36.38695 host node4 0 hdd 7.27739 osd.0 up 1.00000 1.00000 4 hdd 7.27739 osd.4 up 1.00000 1.00000 8 hdd 7.27739 osd.8 up 1.00000 1.00000 12 hdd 7.27739 osd.12 up 1.00000 1.00000 16 hdd 7.27739 osd.16 up 1.00000 1.00000 -9 36.38695 host node5 2 hdd 7.27739 osd.2 up 1.00000 1.00000 6 hdd 7.27739 osd.6 up 1.00000 1.00000 10 hdd 7.27739 osd.10 up 1.00000 1.00000 14 hdd 7.27739 osd.14 up 1.00000 1.00000 18 hdd 7.27739 osd.18 up 1.00000 1.00000 -7 36.38695 host node6 1 hdd 7.27739 osd.1 up 1.00000 1.00000 5 hdd 7.27739 osd.5 up 1.00000 1.00000 9 hdd 7.27739 osd.9 up 1.00000 1.00000 13 hdd 7.27739 osd.13 up 1.00000 1.00000 17 hdd 7.27739 osd.17 up 1.00000 1.00000 -3 29.10956 host node7 3 hdd 7.27739 osd.3 up 1.00000 1.00000 7 hdd 7.27739 osd.7 up 1.00000 1.00000 11 hdd 7.27739 osd.11 up 1.00000 1.00000 15 hdd 7.27739 osd.15 up 1.00000 1.00000 -11 36.38695 host node8 19 hdd 7.27739 osd.19 up 1.00000 1.00000 20 hdd 7.27739 osd.20 up 1.00000 1.00000 21 hdd 7.27739 osd.21 up 1.00000 1.00000 22 hdd 7.27739 osd.22 up 1.00000 1.00000 23 hdd 7.27739 osd.23 up 1.00000 1.00000 [root@node1 ~]# kubectl get pods -n rook-ceph NAME READY STATUS RESTARTS AGE csi-cephfsplugin-9x4dc 3/3 Running 9 14d csi-cephfsplugin-cv7k9 3/3 Running 9 14d csi-cephfsplugin-f6s4k 3/3 Running 22 14d csi-cephfsplugin-fn9w4 3/3 Running 12 14d csi-cephfsplugin-provisioner-59499cbcdd-txtjq 6/6 Running 0 33h csi-cephfsplugin-provisioner-59499cbcdd-wj9dx 6/6 Running 11 4d7h csi-cephfsplugin-tdp99 3/3 Running 9 14d csi-rbdplugin-4bv84 3/3 Running 12 14d csi-rbdplugin-ddgq8 3/3 Running 22 14d csi-rbdplugin-hnnx2 3/3 Running 10 14d csi-rbdplugin-provisioner-857d65496c-qzbcq 6/6 Running 0 33h csi-rbdplugin-provisioner-857d65496c-s6nnd 6/6 Running 9 4d7h csi-rbdplugin-prthz 3/3 Running 9 14d csi-rbdplugin-tjj7z 3/3 Running 12 14d rook-ceph-crashcollector-node4-5bb88d4866-rfv54 1/1 Running 1 4d7h rook-ceph-crashcollector-node5-75dcbb9f45-5tlv8 1/1 Running 0 33h rook-ceph-crashcollector-node6-777c9f579b-zzf69 1/1 Running 0 33h rook-ceph-crashcollector-node7-84464d7cc5-2zthq 1/1 Running 0 33h rook-ceph-crashcollector-node8-66bcdc9c4b-drcj2 1/1 Running 0 3d12h rook-ceph-mgr-a-fb794cf97-cbmws 1/1 Running 26 34h rook-ceph-mon-a-58bf5db667-lhv26 1/1 Running 0 33h rook-ceph-mon-b-78df76b9f8-p7w69 1/1 Running 0 33h rook-ceph-mon-c-6fc84459fd-nc8hx 1/1 Running 0 4d8h rook-ceph-operator-65965c66b5-z5xfl 1/1 Running 2 4d7h rook-ceph-osd-0-59747d74d6-lg5lh 1/1 Running 16 4d7h rook-ceph-osd-1-56c66f7d49-vxj9m 1/1 Running 0 33h rook-ceph-osd-10-6cf5f76d57-g5jhb 1/1 Running 0 33h rook-ceph-osd-11-5446ff487f-qsdmw 1/1 Running 0 33h rook-ceph-osd-12-db4859779-5vrvm 1/1 Running 8 4d7h rook-ceph-osd-13-65d4859986-zx8kx 1/1 Running 0 33h rook-ceph-osd-14-5c6db6c8c9-ghhtk 1/1 Running 0 33h rook-ceph-osd-15-568c6bbf64-m2f2v 1/1 Running 0 33h rook-ceph-osd-16-7d475cc4cc-9lzkp 1/1 Running 8 4d7h rook-ceph-osd-17-6487d4ff79-5x8bh 1/1 Running 0 33h rook-ceph-osd-18-7dd896c985-bjvbz 1/1 Running 0 33h rook-ceph-osd-19-5ccfbf896f-gll6k 1/1 Running 1 4d8h rook-ceph-osd-2-7db5897c7b-fxc4k 1/1 Running 0 33h rook-ceph-osd-20-7b7475f95c-cfprr 1/1 Running 1 4d8h rook-ceph-osd-21-7c544458cf-mfbzr 1/1 Running 0 4d8h rook-ceph-osd-22-76ff6df84-tf4hx 1/1 Running 0 4d8h rook-ceph-osd-23-86b976fdb4-c5xz9 1/1 Running 0 4d8h rook-ceph-osd-3-6ccbd79b9d-dvrwj 1/1 Running 0 33h rook-ceph-osd-4-56c4c85686-pm488 1/1 Running 9 4d7h rook-ceph-osd-5-75dd798967-rmhrp 1/1 Running 0 33h rook-ceph-osd-6-f66fd44fb-f2h2d 1/1 Running 0 33h rook-ceph-osd-7-6fbb57486c-4xvzh 1/1 Running 0 33h rook-ceph-osd-8-59567ccf6-xnmx7 1/1 Running 8 4d7h rook-ceph-osd-9-5b9bc76c98-f8dbf 1/1 Running 0 33h rook-ceph-osd-prepare-node4-jgwvb 0/1 Completed 0 3h1m rook-ceph-osd-prepare-node5-jwvsk 0/1 Completed 0 3h1m rook-ceph-osd-prepare-node6-r2ps9 0/1 Completed 0 3h1m rook-ceph-osd-prepare-node7-cmfsf 0/1 Completed 0 3h1m rook-ceph-osd-prepare-node8-kk256 0/1 Completed 0 3h1m rook-ceph-tools-fc5f9586c-nb2h7 1/1 Running 0 34h

1 0

Kubernetes - How to create a PersistentVolume on an existing durable ceph volume?
by Ralph Soika 12 Jun '21

12 Jun '21

Hi, I have setup a ceph cluster (octopus) and installed he rbd plugins/provisioner in my Kubernetes cluster. I can create dynamically FS and Block Volumes which is fine. For that I have created the following the following storageClass apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph provisioner: rbd.csi.ceph.com parameters: clusterID: <clusterID> pool: kubernetes imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: ceph-system csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: ceph-system csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: ceph-system reclaimPolicy: Delete allowVolumeExpansion: true mountOptions: - discard This works fine for ephemeral dynamically crated volumes. But now I want to use durable volume with the reclaimPolicy:Retain. I expect that I need to create the image in my kubernetes pool on the ceph cluster first - which I have done. I defined the following new storage class with the reclaimPolicy 'Retain': apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-durable provisioner: rbd.csi.ceph.com parameters: clusterID: <clusterID> pool: kubernetes imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: ceph-system csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: ceph-system csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: ceph-system reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - discard And finally I created the following PersistentVolume and PersistentVolumeClaim: --- kind: PersistentVolume apiVersion: v1 metadata: name: demo-internal-index spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: - ReadWriteOnce claimRef: namespace: office-demo-internal name: index csi: driver: driver.ceph.io fsType: ext4 volumeHandle: demo-internal-index storageClassName: ceph-durable --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: index namespace: office-demo-internal spec: accessModes: - ReadWriteOnce storageClassName: ceph-durable resources: requests: storage: 1Gi volumeName: "demo-internal-index" But this seems not to work and I can see the following deployment warning: attachdetach-controller AttachVolume.Attach failed for volume "demo-internal-index" : attachdetachment timeout for volume demo-internal-index But the PV exists: $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE demo-internal-index 1Gi RWO Retain Bound office-demo-internal/index ceph-durable 2m35s and also the PVC exists: $ kubectl get pvc -n office-demo-internal NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE index Bound demo-internal-index 1Gi RWO ceph-durable 53m I guess my PV object is nonsense? Can someone provide me an example how to setup the PV object in Kubernetes. I only found examples where the Ceph Monitoring IPs and the user/password is configured within the PV object. But I would expect that this is covered by the storage class already? Thanks for your help === Ralph

1 1

Error on Ceph Dashboard
by Robert W. Eckert 11 Jun '21

11 Jun '21

Hi - this just started happening in the past few days using Ceph Pacific 16.2.4 via cephadmin (Podman containers) The dashboard is returning No active ceph-mgr instance is currently running the dashboard. A failover may be in progress. Retrying in 5 seconds... And ceph status returns cluster: id: fe3a7cb0-69ca-11eb-8d45-c86000d08867 health: HEALTH_WARN Module 'dashboard' has failed dependency: cannot import name 'AuthManager' clock skew detected on mon.cube services: mon: 3 daemons, quorum story,cube,rhel1 (age 46h) mgr: cube.tvlgnp(active, since 47h), standbys: rhel1.zpzsjc, story.gffann mds: 2/2 daemons up, 1 standby osd: 13 osds: 13 up (since 46h), 13 in (since 46h) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 497 pgs objects: 1.50M objects, 2.1 TiB usage: 6.2 TiB used, 32 TiB / 38 TiB avail pgs: 497 active+clean io: client: 255 B/s rd, 2.7 KiB/s wr, 0 op/s rd, 0 op/s wr The only thing that has happened on the cluster was one of the servers was rebooted. No configuration changes were performed Any suggestions? Thanks, rob

2 3

driver name rbd.csi.ceph.com not found in the list of registered CSI drivers ?
by Ralph Soika 11 Jun '21

11 Jun '21

Hi, I try to connect my new ceph cluster (octopus) with my kubernetes system. Therefor I followed the setup guide form the official documentation: https://docs.ceph.com/en/octopus/rbd/rbd-kubernetes/ The csi-rbdplugin-provisioner is running successful on all my kubernetes worker nodes (as far as I can see). Now I try to deploy the nginx example : --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ceph-pvc spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 1Gi storageClassName: ceph --- apiVersion: v1 kind: Pod metadata: name: csi-rbd-demo-pod spec: containers: - name: web-server image: nginx volumeMounts: - name: mypvc mountPath: /var/lib/www/html volumes: - name: mypvc persistentVolumeClaim: claimName: ceph-pvc readOnly: false A Persitence volume is created $ kubectl get pv .... pvc-5c64ed45-adde-4fe7-9b38-9d4c7a8f7d34 1Gi RWO Delete Bound default/ceph-pvc ceph 7m15s and also the persitence volume claim: $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-pvc Bound pvc-5c64ed45-adde-4fe7-9b38-9d4c7a8f7d34 1Gi RWO ceph 8m14s But the POD is not deployed, because of the following error message: MountVolume.MountDevice failed for volume "pvc-5c64ed45-adde-4fe7-9b38-9d4c7a8f7d34" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rbd.csi.ceph.com not found in the list of registered CSI drivers Can someone help me to understand the meaning of this error message? Did I need to install something else ? Maybe a ceph-fs-plugin...? Thanks for any help === Ralph --

1 1

suggestion for Ceph client network config
by Götz Reinicke 11 Jun '21

11 Jun '21

Hi all We get a new samba smb fileserver who mounts our cephfs for exporting some shares. What might be a good or better network setup for that server? Should I configure two interfaces - one for the smb share export towards our workstations and desktops and one towards the ceph cluster? Or would it be „ok“ for all traffic to be on one interface? The server has 40G ports. Thanks for your suggestions and feedback . Regards . Götz

2 1

lib remoto in ubuntu
by Alfredo Rezinovsky 11 Jun '21

11 Jun '21

I cannot enable cephadm because it cannot find remoto lib. Even when I installed it using "pip3 install remoto" and then installed ir from the deb package build from the git sources at https://github.com/alfredodeza/remoto/ If I type "import remoto" in a python3 prompt it works. -- Alfrenovsky

2 1

Ceph Ansible fails on check if monitor initial keyring already exists
by Jared Jacob 11 Jun '21

11 Jun '21

I am running the Ceph ansible script to install ceph version Stable-6.0 (Pacific). When running the sample yml file that was supplied by the github repo it runs fine up until the "ceph-mon : check if monitor initial keyring already exists" step. There it will hang for 30-40 minutes before failing. From my understanding ceph ansible should be creating this keyring and using it for communication between monitors, so does anyone know why the playbook would have a hard time with this step? Thanks in advance!

2 1

ceph and openstack throttling experience
by Marcel Kuiper 10 Jun '21

10 Jun '21

Hi We're running ceph nautilus 14.2.21 (going to octopus latest in a few weeks) as volume and instance backend for our openstack vm's. Our clusters run somewhere between 500 - 1000 OSDs on SAS HDDs with NVMe's as journal and db device Currently we do not have our vm's capped on iops and throughput. We regularly get slowops warnings (once or twice per day) and wonder whether there are more users with sort of the same setup that do throttle their openstack vm's. - What kind of numbers are used in the field for IOPS and throughput limiting? - As a side question, is there an easy way to get rid of the slowops warning besides restarting the involved osd. Otherwise the warning seems to stay forever Regards Marcel

2 3

Ceph Octopus - How to customize the Grafana configuration
by Ralph Soika 10 Jun '21

10 Jun '21

Hello, I have installed and bootsraped a Ceph manager node via cephadm and the options: --initial-dashboard-user admin --initial-dashboard-password [PASSWORD] --dashboard-password-noupdate Everything works fine. I also have the Grafana Board to monitor my cluster. But the access to Grafana is open for anonymous users because of the grafana.ini template with the option: [auth.anonymous] enabled = true I can't figure out how to tweak the default grafana.ini file. Can someone help me how to do this? I tried to do this with the command: # ceph config-key set mgr/cephadm/services/grafana/grafana.ini \ -i /tmp//grafana.ini.j2 # ceph orch reconfig grafana But without any effect. I also did not really understand where I should place the grafana.ini file on my Host? Thanks for any help === Ralph --

2 3

nautilus: rbd ls returns ENOENT for some images
by Peter Lieven 10 Jun '21

10 Jun '21

Hi, we currently run into an issue where a rbd ls for a namespace returns ENOENT for some of the images in that namespace. /usr/bin/rbd --conf=XXX --id XXX ls 'mypool/28ef9470-76eb-4f77-bc1b-99077764ff7c' -l --format=json 2021-06-09 11:03:34.916 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55cacccc2390 fail: (2) No such file or directory 2021-06-09 11:03:34.916 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55caccd2b920 fail: (2) No such file or directory 2021-06-09 11:03:34.920 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55caccd9b4e0 fail: (2) No such file or directory rbd: error opening 34810ac2-3112-4fef-938c-b76338b0eeaf.raw: (2) No such file or directory rbd: error opening c9882583-6dd5-4eca-bb82-3e81f7d63fa9.raw: (2) No such file or directory rbd: error opening 5d5251d1-f017-4382-845c-65e504683742.raw: (2) No such file or directory 2021-06-09 11:03:34.924 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55cacce07b00 fail: (2) No such file or directory rbd: error opening c625b898-ec34-4446-9455-d2b70d9e378f.raw: (2) No such file or directory 2021-06-09 11:03:34.924 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55caccd7cce0 fail: (2) No such file or directory rbd: error opening 990c4bbe-6a7b-4adf-aab8-432e18d79e58.raw: (2) No such file or directory 2021-06-09 11:03:34.924 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55cacce336f0 fail: (2) No such file or directory rbd: error opening 7382eb5b-a3eb-41e2-89b6-512f7b1d86c0.raw: (2) No such file or directory [{"image":"108600c6-2312-4d61-9f5b-35b351112512.raw","size":31457280000,"format":2,"lock_type":"exclusive"},{"image":"1292ef0c-2333-44f1-be30-39105f7d176e.raw","size":262149242880,"format":2,"lock_type":"exclusive"},{"image":"8cda5c3f-cdbd-42f4-918f-1480354e7965.raw","size":262149242880,"format":2,"lock_type":"exclusive"}] rbd: listing images failed: (2) No such file or directory The way to trigger this state was that the images which show "No such file or directory" were deleted with rbd rm, but the operation was interrupted (rbd process was killed) due to a timeout. What is the best way to recover from this and how to properly clean up? Release is nautilus 14.2.20 Thanks, Peter

2 4

Integration of openstack to ceph
by Michel Niyoyita 10 Jun '21

10 Jun '21

Dear Ceph Users, Anyone can help on the guidance of how I can integrate ceph to openstack ? especially RGW. Regards Michel

3 2

delete stray OSD daemon after replacing disk
by mabi 10 Jun '21

10 Jun '21

Hello, I replaced an OSD disk on one of my Nautilus OSD node which created a new osd number. Now ceph shows that there is one cephadm stray daemon (the old OSD #1 which I replaced) and which I can't remove as you can see below: # ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.1 on host ceph1e not managed by cephadm # ceph orch daemon rm osd.1 --force Error EINVAL: Unable to find daemon(s) ['osd.1'] Is there another command I am missing? Best regards, Mabi

3 4

1 daemons have recently crashed
by feng.zhang＠orcadt.com 10 Jun '21

10 Jun '21

Hi； ceph -s HEALTH_WARN [root@node-1 ~]# ceph crash info 2021-06-07T10:40:07.248249Z_c98563f9-78ef-4b9f-9f6b-816d2d2a333d { "backtrace": [ "(()+0xf5f0) [0x7fbb55b8f5f0]", "(std::_Rb_tree<boost::intrusive_ptr<AsyncConnection>, boost::intrusive_ptr<AsyncConnection>, std::_Identity<boost::intrusive_ptr<AsyncConnection> >, std::less<boost::intrusive_ptr<AsyncConnection> >, std::allocator<boost::intrusive_ptr<AsyncConnection> > >::find(boost::intrusive_ptr<AsyncConnection> const&) const+0x24) [0x556347e234b4]", "(AsyncConnection::_stop()+0xa7) [0x556347e1da87]", "(ProtocolV2::stop()+0x88) [0x556347e44e78]", "(ProtocolV2::_fault()+0x65) [0x556347e44fe5]", "(ProtocolV2::handle_read_frame_preamble_main(std::unique_ptr<ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer>&&, int)+0x30a) [0x556347e5ad5a]", "(ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x34) [0x556347e45b44]", "(AsyncConnection::process()+0x794) [0x556347e208e4]", "(EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa25) [0x556347c819d5]", "(()+0xd3c3e7) [0x556347c873e7]", "(()+0x1224c5f) [0x55634816fc5f]", "(()+0x7e65) [0x7fbb55b87e65]", "(clone()+0x6d) [0x7fbb54a4b88d]" ], "ceph_version": "2.0.11", "crash_id": "2021-06-07T10:40:07.248249Z_c98563f9-78ef-4b9f-9f6b-816d2d2a333d", "entity_name": "osd.45", "os_id": "centos", "os_name": "CentOS Linux", "os_version": "7 (Core)", "os_version_id": "7", "process_name": "ceph-osd", "stack_sig": "897fe7f6bf2184fafd5b8a29905a147cb66850db318f6e874292a278aeb615bb", "timestamp": "2021-06-07T10:40:07.248249Z", "utsname_hostname": "node-8", "utsname_machine": "x86_64", "utsname_release": "5.4.22-1.el7.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Mon Mar 30 13:15:02 EDT 2020" } [root@node-1 ~]# Best Regards！ -------------------------------------- zhangfeng --------------------------------------- 西安奥卡云数据科技有限公司 ORCA DATA TECHNOLOGY (XI'AN) CO., LTD Email:feng.zhang@orcadt.com Phone: 13992881878 website: https://www.orcadt.com

1 0

Performance (RBD) regression after upgrading beyond v15.2.8
by Wido den Hollander 09 Jun '21

09 Jun '21

Hi, While doing some benchmarks I have two identical Ceph clusters: 3x SuperMicro 1U AMD Epyc 7302P 16C 256GB DDR 4x Samsung PM983 1,92TB 100Gbit networking I tested on such a setup with v16.2.4 with fio: bs=4k qd=1 IOps: 695 That was very low as I was expecting at least >1000 IOps. I checked with the second Ceph cluster which was still running v15.2.8, the result: 1364 IOps. I then upgraded from 15.2.8 to 15.2.13: 725 IOps Looking at the differences between v15.2.8 and v15.2.8 of options.cc I saw these options: bluefs_buffered_io: false -> true bluestore_cache_trim_max_skip_pinned: 1000 -> 64 The main difference seems to be 'bluefs_buffered_io', but in both cases this was already explicitly set to 'true'. So anything beyond 15.2.8 is right now giving me a much lower I/O performance with Queue Depth = 1 and Block Size = 4k. 15.2.8: 1364 IOps 15.2.13: 725 IOps 16.2.4: 695 IOps Has anybody else seen this as well? I'm trying to figure out where this is going wrong. Wido

3 3

Is it safe to mix Octopus and Pacific mons?
by Vladimir Brik 09 Jun '21

09 Jun '21

Hello My attempt to upgrade from Octopus to Pacific ran into issues, and I currently have one 16.2.4 mon and two 15.2.12 mons. Is this safe to run the cluster like this or should I shut down the 16.2.4 mon until I figure out what to do next with the upgrade? Thanks, Vlad

2 1

Upgrade to 16 failed: wrong /sys/fs/cgroup path
by Vladimir Brik 09 Jun '21

09 Jun '21

Hello My upgrade from 15.2.12 to 16.2.4 is stuck because a mon daemon failed to upgrade. Systemctl status of the mon showed this error: Error: open /sys/fs/cgroup/cpuacct,cpu/system.slice/... It turns out there is no /sys/fs/cgroup/cpuacct,cpu directory on my system. Instead, I have /sys/fs/cgroup/cpu,cpuacct. Symlinking them appears to have solved the immediate problem, but if I proceed with the upgrade to 16.2.4, after reboot, all ceph daemons will probably fail to start. Is this an issue with ceph, podman (2.1.1), or the fact that I am running Centos7? Is it possible to upgrade from 15.2.12 to 16.2.4 on Centos7? I thought that installing a version of podman that is compatible with both would suffice, but apparently not... Vlad

1 0

omap sizes
by Szabo, Istvan (Agoda) 09 Jun '21

09 Jun '21

Hi, Is there a way to check the omap sizes in the index pool? Either key numbers and size also? This one doesn't work: https://ceph.com/geen-categorie/get-omap-keyvalue-size/ Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Ceph Ansible fails on check if monitor initial keyring already exists
by Jared Jacob 08 Jun '21

08 Jun '21

I am running the Ceph ansible script to install ceph version Stable-6.0 (Pacific). When running the sample yml file that was supplied by the github repo it runs fine up until the "ceph-mon : check if monitor initial keyring already exists" step. There it will hang for 30-40 minutes before failing. From my understanding ceph ansible should be creating this keyring and using it for communication between monitors, so does anyone know why the playbook would have a hard time with this step? Thanks in advance!

1 0

DocuBetter Meeting -- 09 June 2021 1730 UTC
by John Zachary Dover 08 Jun '21

08 Jun '21

A DocuBetter Meeting will be held on 09 June 2021 at 1730 UTC. This is the monthly DocuBetter Meeting that is more convenient for European and North American Ceph contributors than the other meeting, which is convenient for people in Australia and Asia (and which is very rarely attended). Topics: - cephadm docs rewrite (ongoing) - ceph.io copy rewrite and information architecture restructure (ongoing) - rgw manual install procedure (prospective) - new ceph-docs mailing list Bring your docs complaints and requests to this meeting. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

Announcing go-ceph v0.10.0
by John Mulligan 08 Jun '21

08 Jun '21

I'm happy to announce another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.10.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. In addition to our regular release this week, we're also participating in this June's "Ceph Month" event with the "go-ceph get together" Birds-of-a-Feather session on Thursday June 10th at 10:10 Eastern time. It should be visible in the Ceph Community calendar [1]. If you can't make the BoF, questions, comments, bugs etc are best directed at our github issues tracker or github discussions forum. [1] - https://ceph.io/contribute/#community-calendar -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

1 0

ceph buckets
by Rok Jaklič 08 Jun '21

08 Jun '21

Hi, I try to create buckets through rgw in following order: - *bucket1* with *user1* with *access_key1* and *secret_key1* - *bucket1* with *user2* with *access_key2* and *secret_key2* when I try to create a second bucket1 with user2 I get *Error response code BucketAlreadyExists.* Why? Should not buckets relate only to users? Is this by design and is there any particular reason that it follows this concept? Regards, Rok

3 4

Bucket creation on RGW Multisite env.
by mhnx 08 Jun '21

08 Jun '21

Hello. I have a multisite RGW environment. When I create a new bucket, the bucket is directly created on master and secondary. If I don't want to sync a bucket, I need to stop sync after creation. Is there any global option as "Do not sync directly, only start if I want to" ?

3 6

Index pool hasn't been cleaned up and caused large omap, safe to delete the index file?
by Szabo, Istvan (Agoda) 08 Jun '21

08 Jun '21

Hi, In my multisite setup 1 big bucket has been deleted and seems like hasn't been cleaned up on one of the secondary site. Is it safe to delete the 11 shard objects from the index pool which holding the omaps of that bucket files? Also a quick question, is it a problem if we use like this? Create a bucket which means create in all dc Don't create any bucket sync the user upload different files in different dcs. When a bucket deletion happens would this usage behavior cause issue that different files are in the buckets? If yes, how to prevent this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 1

Re: OT: How to Build a poor man's storage with ceph
by Sebastian Knust 08 Jun '21

08 Jun '21

Hi Michael, On 08.06.21 11:38, Ml Ml wrote: > Now i was asked if i could also build a cheap 200-500TB Cluster > Storage, which should also scale. Just for Data Storage such as > NextCloud/OwnCloud. With similar requirements (server primarily for Samba and NextCloud, some RBD use, very limited budget) I am using HDD for data and SSD for system and CephFS metadata. Note that I am running NextCloud on CephFS storage. If you want to go with RGW/S3 as a storage backend instead, the following might not apply to your use case. My nodes (bought end of 2020) are: - 2U chassis with 12 3.5" SATA slots - Intel Xeon Silver 4208 - 128 GB RAM - 2 x 480 GB Samsung PM883 SSD -> 50 GB in MD-RAID1 for system -> 430 GB OSD (one per SSD) - initially 6 x 14 TB Enterprise HDD - 4 x 10 GBase-T (active/passive bonded, dedicated backend network) Each node with this configuration cost about 4k EUR net at the end of 2020. Due to increasing prices for storage, it will be a bit more expensive now. I am running five nodes now and have added a few more disks (ranging 8-14 TB), nearly filling up the nodes. My experience so far: - I had to throttle scrubbing (see below for details) - For purely NextCloud and Samba performance is sufficient for a few hundred concurrent users with a handful of power users - Migration of the mail server to this cluster was a disaster due to limited IOPS, had to add some more SSDs and place the mail server in an SSD-only pool. - MDS needs a lot of memory for larger CephFS installs, I will move it to a dedicated server probably next year. 128 GB per node works but I would not recommend any less. - Rebalancing takes an eternity (2-3 weeks), so make sure that your PG nums are okay from the start - I have all but given up on snapshots with CephFS due to severe performance degradation with kernel client during backup My scrubbing config looks like this: osd_backfill_scan_max 16 osd_backfill_scan_min 4 osd_deep_scrub_interval 2592000.000000 osd_deep_scrub_randomize_ratio 0.030000 osd_recovery_max_active_hdd 1 osd_recovery_max_active_ssd 5 osd_recovery_sleep_hdd 0.050000 osd_scrub_begin_hour 18 osd_scrub_end_hour 7 osd_scrub_chunk_max 1 osd_scrub_chunk_min 1 osd_scrub_max_interval 2419200.000000 osd_scrub_min_interval 172800.000000 osd_scrub_sleep 0.100000 My data is in a replicated pool with n=3 without compression. You might also consider EC and then want to aim for more nodes. Cheers Sebastian

1 0

OT: How to Build a poor man's storage with ceph
by Ml Ml 08 Jun '21

08 Jun '21

Hello List, i used to build 3 Node Cluster with spinning Rust and later with (Enterprise) SSDs. All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks and i was done. The Requirements were just 10/15TB Disk usage (30-45TB Raw). Now i was asked if i could also build a cheap 200-500TB Cluster Storage, which should also scale. Just for Data Storage such as NextCloud/OwnCloud. Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x 45k EUR = 135k EUR. Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD) How do the "big boys" do this? Just throw money at it? Would a mix of OSD SSD Metadata + Spinning Rust do the job? My experience so far is that each time i had a crash/problem it was always such a pain to wait for the spinning rust. Do you have any experience/hints on this? Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which would speed up the performance, but have a bigger impact on a dying disk. My requirements are more or less low IO Traffic but loads of disk space usage. Any hints/ideas/links are welcome. Cheers, Michael

4 3

How to enable lazyio under kcephfs?
by opengers 08 Jun '21

08 Jun '21

ceph: 14.2.x kernel: 4.15 In cephfs, due to the need for cache consistency, When a client is executing buffer IO, another client will hang when reading and writing the same file It seems that lazyio can solve this problem, lazyio allows multiple clients to execute buffer IO at the same time(relax consistency), But I am not sure how to enable lazyio under the kernel mount, the test found that the "client_force_lazyio" parameter does not work My final requirement is to use lazyio to implement multiple clients to read and write the same file(buffer IO mode) Can someone explain how to enable lazyio under kcephfs, thanks

2 1

Only 2/5 mon services running
by Jeremy Hansen 08 Jun '21

08 Jun '21

In an attempt to troubleshoot why only 2/5 mon services were running, I believe I’ve broke something: [ceph: root@cn01 /]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager 1/1 81s ago 9d count:1 crash 6/6 7m ago 9d * grafana 1/1 80s ago 9d count:1 mds.testfs 2/2 81s ago 9d cn01.ceph.la1.clx.corp;cn02.ceph.la1.clx.corp;cn03.ceph.la1.clx.corp;cn04.ceph.la1.clx.corp;cn05.ceph.la1.clx.corp;cn06.ceph.la1.clx.corp;count:2 mgr 2/2 81s ago 9d count:2 mon 2/5 81s ago 9d count:5 node-exporter 6/6 7m ago 9d * osd.all-available-devices 20/26 7m ago 9d * osd.unmanaged 7/7 7m ago - <unmanaged> prometheus 2/2 80s ago 9d count:2 I tried to stop and start the mon service, but now the cluster is pretty much unresponsive, I’m assuming because I stopped mon: [ceph: root@cn01 /]# ceph orch stop mon Scheduled to stop mon.cn01 on host 'cn01.ceph.la1.clx.corp' Scheduled to stop mon.cn02 on host 'cn02.ceph.la1.clx.corp' Scheduled to stop mon.cn03 on host 'cn03.ceph.la1.clx.corp' Scheduled to stop mon.cn04 on host 'cn04.ceph.la1.clx.corp' Scheduled to stop mon.cn05 on host 'cn05.ceph.la1.clx.corp' [ceph: root@cn01 /]# ceph orch start mon ^CCluster connection aborted Now even after a reboot of the cluster, it’s unresponsive. How do I get mon started again? I’m going through Ceph and breaking things left and right, so I apologize for all the questions. I learn best from breaking things and figuring out how to resolve the issues. Thank you -jeremy

1 1

Global Recovery Event
by Jeremy Hansen 08 Jun '21

08 Jun '21

I’m seeing this in my health status: progress: Global Recovery Event (13h) [............................] (remaining: 5w) I’m not sure how this was initiated but this is a cluster with almost zero objects. Is there a way to halt this process? Why would it estimate 5 weeks to recover a cluster with almost zero data? [ceph: root@cn01 /]# ceph -s -w cluster: id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d health: HEALTH_OK services: mon: 2 daemons, quorum cn02,cn05 (age 13h) mgr: cn01.ceph.la1.clx.corp.xnkoft(active, since 13h), standbys: cn02.arszct mds: 1/1 daemons up, 1 standby osd: 27 osds: 27 up (since 13h), 27 in (since 16h) data: volumes: 1/1 healthy pools: 3 pools, 65 pgs objects: 22.09k objects, 86 GiB usage: 261 GiB used, 98 TiB / 98 TiB avail pgs: 65 active+clean progress: Global Recovery Event (13h) [............................] (remaining: 5w) Thanks -jeremy

2 2

Failed OSD has 29 Slow MDS Ops.
by Dave Hall 07 Jun '21

07 Jun '21

Hello, Nautilus 14.2.16 I had an OSD go bad about 10 days ago. Apparently as it was going down some MDS ops got hung up waiting for it to come back. I was out of town for a couple days and found the OSD 'Down and Out' when I checked in. (Also, oddly, the cluster did not appear to initiate recovery right away - it took until I rebooted the OSD node.) As of right now, the damaged OSD is 'safe-to-destroy' but the slow ops are still hanging around. Earlier today I quiesced the clients that were accessing the CephFS, then unmounted and re-mounted it. However, this did not clear the lingering ops. When I had the node offline I verified that the HDD and NVMe associated with the OSD seem to actually be healthy, so I plan to zap and re-deploy using the same hardware. I would also like to upgrade to 14.2.20 (latest Ceph for debian 10), but I'm hesitant to do any of this until I get rid of these 29 slow ops. Can anybody suggest a path forward? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

Connect ceph to proxmox
by Szabo, Istvan (Agoda) 07 Jun '21

07 Jun '21

Hi, Is there a way to connect from my nautilus ceph setup the pool that I created in ceph to proxmox? Or need a totally different ceph install? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 5

Re: Why you might want packages not containers for Ceph deployments
by Marc 07 Jun '21

07 Jun '21

Do you use rbd images in containers that are residing on osd nodes? Does this give any problems? I used to have kernel mounted cephfs on a osd node, after a specific luminous release this was giving me problems. > -----Original Message----- > From: Eneko Lacunza <elacunza(a)binovo.es> > Sent: Friday, 4 June 2021 15:49 > To: ceph-users(a)ceph.io > Subject: *****SPAM***** [ceph-users] Re: Why you might want packages > not containers for Ceph deployments > > Hi, > > We operate a few Ceph hyperconverged clusters with Proxmox, that > provides a custom ceph package repository. They do a great work; and > deployment is a brezee. > > So, even as currently we would rely on Proxmox packages/distribution and > not upstream, we have a number of other projects deployed with > containers and we even distribute some of our own development in deb and > container packages, so I will comment on our view: > > El 2/6/21 a las 23:26, Oliver Freyermuth escribió: > [...] > > > > If I operate services in containers built by developers, of course > > this ensures the setup works, and dependencies are well tested, and > > even upgrades work well — but it also means that, > > at the end of the day, if I run 50 services in 50 different containers > > from 50 different upstreams, I'll have up to 50 different versions of > > OpenSSL floating around my production servers. > > If a security issue is found in any of the packages used in all the > > container images, I now need to trust the security teams of all the 50 > > developer groups building these containers > > (and most FOSS projects won't have the ressources, understandably...), > > instead of the one security team of the disto I use. And then, I also > > have to re-pull all these containers, after finding out that a > > security fix has become available. > > Or I need to build all these containers myself, and effectively take > > over the complete job, and have my own security team. > > > > This may scale somewhat well, if you have a team of 50 people, and > > every person takes care of one service. Containers are often your > > friend in this case[1], > > since it allows to isolate the different responsibilities along with > > the service. > > > > But this is rarely the case outside of industry, and especially not in > > academics. > > So the approach we chose for us is to have one common OS everywhere, > > and automate all of our deployment and configuration management with > > Puppet. > > Of course, that puts is in one of the many corners out there, but it > > scales extremely well to all services we operate, > > and I can still trust the distro maintainers to keep the base OS safe > > on all our servers, automate reboots etc. > > > > For Ceph, we've actually seen questions about security issues already > > on the list[0] (never answered AFAICT). > > These are the two main issues I find with containers really: > > - Keeping hosts uptodate is more complex (apt-get update+apt-get > dist-upgrade and also some kind of docker pull+docker > restart/docker-compose up ...). Much of the time the second part is not > standard (just deployed a Harbor service, upgrade is quite simple but I > have to know how to do it as it's speciffic, maintenance would be much > easier if it was packaged in Debian). I won't say it's more difficult, > but it will be more diverse and complex. > > - Container image quality and security support quality, that will vary > from upstream to upstream. You have to research each of them to know > were they stand. A distro (specially a good one like Debian, Ubuntu, > RHEL or SUSE) has known, quality security support for the repositories. > They will even fix issues not fixed by upstream (o backport them to > distro's version...). This is more an upstream vs distro issue, really. > > About debugging issues reported with Ceph containers, I think those are > things waiting for a fix: why are logs writen in container image (or an > ephemeral volume, I don't know really how is that done right now) > instead of an external name volume o a local mapped dir in /var/log/ceph ? > > All that said, I think that it makes sense for an upstream project like > Ceph, to distribute container images, as it is the most generic way to > distribute (you can deploy on any system/distro supporting container > images) and eases development. But only distributing container images > could make more users depend on third party distribution (global or > specific distros), which would delay feeback/bugreport to upstream. > > Cheers and thanks for the great work! > > Eneko Lacunza > Zuzendari teknikoa | Director técnico > Binovo IT Human Project > > Tel. +34 943 569 206 | https://www.binovo.es > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun > > https://www.youtube.com/user/CANALBINOVO > https://www.linkedin.com/company/37269706/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Turning on "compression_algorithm" old pool with 500TB usage
by mhnx 07 Jun '21

07 Jun '21

Hello. I have a erasure pool and I didn't turn on compression at the beginning. Now I'm writing new type of very small data and overhead is becoming an issue. I'm thinking to turn on compression on the pool but in most filesystems it will effect only the new data. What is the behavior in ceph?

3 2

Cephfs root/boot?
by Harry G. Coin 07 Jun '21

07 Jun '21

Has anyone added the 'conf.d' modules (and on the centos/rhel/fedora world done the selinux work) so that initramfs/dracut can 'direct kernel boot' cephfs as a guest image root file system? It took some work for the nfs folks to manage being the root filesystem. Harry

1 0

Running ISCSI with Ubuntu 18.04 OS
by Michel Niyoyita 07 Jun '21

07 Jun '21

Dear all , IS it possible to configure and run ISCSI when you are deploying ceph using ansible running on ubuntu 18.04 OS? please help me to know and if possible provide helpful links on that. Best Regard Michel

1 0

cephfs objets without 'parent' xattr?
by Burkhard Linke 07 Jun '21

07 Jun '21

Hi, during an OS upgrade from Ubuntu 18.04 to 20.04 we seem to have triggered a bcache bug on three OSD hosts. These hosts are used with a 6+2 EC pool used with CephFS, so a number of PGs are affected by the bug. We were able to restart two of the three hosts (and will run some extra scrubs on all PGs), but at least 7 PGs have unfound objects now. I'm currently trying to find out which files are affected to restore them from backup or inform the users about data corruption for files in dedicated scratch directories. This works quite well for most of the files. The 'parent' xattr attached to the file's first chunks contains the complete path of the file within the filesystem, so locating the files should be easy with a little help from ceph-dencoder. But there are some files that do not have the 'parent' xattr: for pg in $(ceph health detail | grep active\+recovery_unfound | cut -d' ' -f 6); do echo $pg; for obj in $(ceph pg $pg list_unfound | jq -r '.objects | .[] | .oid.oid'| cut -c1-11); do rados -p bcf_fs_data_rep getxattr $obj.00000000 parent > $obj.parent; done; done gives 77.1 error getting xattr bcf_fs_data_rep/1002a143927.00000000/parent: (2) No such file or directory ..... 'bcf_fs_data_rep' is the first data pool of the filesystem which should contain the xattr data. But for a number of objects (58 of 928) the above command is not able to retrieve the information. Questions: 1. If the 'parent' xattr is not available in the first data pool (and neither in the meta data pool), what might be the state of these objects? Can they be deleted by using 'mark_unfound delete'? 2. the list_unfound command only prints 256 objects; how can this limit be lifted, since some pools have more unfound objects? Regards, Burkhard

1 0

Debian buster nautilus 14.2.21 missing?
by Szabo, Istvan (Agoda) 07 Jun '21

07 Jun '21

Hi, Am I doing something wrong or the 21 update is missing for buster? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
by Jeremy Hansen 07 Jun '21

07 Jun '21

What’s the proper way to track down where this error is coming from? Thanks. 6/7/21 12:40:00 AM [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 6/7/21 12:40:00 AM [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s)

2 4

Re: SSD recommendations for RBD and VM's
by mhnx 06 Jun '21

06 Jun '21

Hello Samuel. Thanks for the answer. Yes the Intel S4510 series is a good choice but it's expensive. I have 21 server and data distribution is quite well. At power loss I don't think I'll lose data. All the VM's using same image and the rest is cookie. In this case I'm not sure I should spend extra money on PLP. Actually I like Samsung 870 EVO. It's cheap and I think 300TBW will be enough for 5-10years. Do you know any better ssd with the same price range as 870 EVO? Samsung 870 evo (500GB) = 5 Years or 300 TBW - $64.99 Samsung 860 pro (512GB) = 5 Years or 600 TBW - $99 > > I would recommend Intel S4510 series, which has power loss protection (PLP). > > If you do not care about PLP, lower-cost Samsung 870EVO and Crucial MX500 should also be OK (with separate DB/WAL on enterprise SSD with PLP) > > Samuel > > ________________________________ > huxiaoyu(a)horebdata.cn > > > From: by morphin > Date: 2021-05-30 02:48 > To: Anthony D'Atri > CC: Ceph Users > Subject: [ceph-users] Re: SSD recommendations for RBD and VM's > Hello Anthony. > > I use Qemu and I don't need size. > I've 1000 vm and usually they're clones from the same rbd image. The > image is 30GB. > Right now I've 7TB Stored data. rep x3 = 20TB data. It's mostly read > intensive. Usage is stable and does not grow. > So I need I/O more than capacity. That's why I'm looking for 256-512GB SSD's. > I think right now 480-512GB is sweet spot for $ / GB. So 60PCS 512GB > will be enough. Actually 120PCS 256GB will be better but the price > goes up. > I have Dell R720-740 and I use SATA Intel DCS3700 for journal. I've > 40PCS 100GB. I'm gonna make them OSD as well. > 7 years and DC S3700 still rocks. Not even one of them is dead. > The SSD must be Low price & High TBW life span. Rest is not important. > > > Anthony D'Atri <anthony.datri(a)gmail.com>, 30 May 2021 Paz, 02:26 > tarihinde şunu yazdı: > > > > The choice depends on scale, your choice of chassis / form factor, budget, workload and needs. > > > > The sizes you list seem awfully small. Tell us more about your use-case. OpenStack? Proxmox? QEMU? VMware? Converged? Dedicated ? > > —aad > > > > > > > On May 29, 2021, at 2:10 PM, by morphin <morphinwithyou(a)gmail.com> wrote: > > > > > > Hello. > > > > > > I have virtualization env and I'm looking new SSD for HDD replacement. > > > What are the best Performance / Price SSDs in the market right now? > > > I'm looking 1TB, 512GB, 480GB, 256GB, 240GB. > > > > > > Is there a SSD recommendation list for ceph? > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

5 9

RBD + ZFS + NFS = bad performance. How to speed up?
by mhnx 05 Jun '21

05 Jun '21

Hello. I'm using rbd disks for zfs and on the zfs I use NFS. I see 10-50MB/s top. When I use the rbd disk in the VM environment it can reach 1GB/s RW so there is nothing wrong with the rbd and the cluster. Also I'm sure about ZFS+NFS performance when zfs on the real device. But when they work together I can't get even %10 performance. I think something is wrong, it shouldn't be that bad. Ceph version: Nautilus 14.2.16 RBD Data = EC Pool RBD Metadata = SSD replicated zpool create $testpool rbd0 exportfs /testpool Is there anyone who has tried it before?

1 1

Zabbix sender issue
by Bob Loi 04 Jun '21

04 Jun '21

Hi, I managed to build a Ceph cluster with the help of cephadm tool. It works like a charm. I have a problem that i’m still not able to fix: I know that zabbix-sender executable is not integrated into the cephadm image of ceph-mgr pulled and started by podman because of this choice. https://github.com/ceph/ceph-container/issues/1651 I’m a total newbie about containers tecnologies, but i managed to install zabbix sender manually by executing this with podman podman ps -a -> looking for the container ID podman exec -ti [id_container] /bin/bash -> to Then install repo via rpm and dnf install zabbix-sender. Two considerations. 1) https://github.com/ceph/ceph-container/issues/1651 -> This answer still leaves me confused. It’s pretty no-sense giving a Zabbix module if then you have to install the executable by yourself and the container it’s overwritten everytime you reboot the physical host giving minimal info about avoiding this behavior. I know this is open source and you have to know well the environment before doing everything, i know devs want less troubles as possible, but this is about user experience. If you have to choose between the final user or you having an annoying problem, IMHO, i prefer to have a happy user. End of consideration. I think including zabbix-sender executable into the container wouldn’t kill nobody. 2) Since i’m a total noob about containers, podman, docker etc, do you have any info about "fix" this behavior and avoid overwrite the mgr containers everytime i reboot the host? Please forgive me, i’m totally a newbie about containers. Thank you in advance for the support. Best, Roberto Kirecom.net

1 0

Rolling upgrade model to new OS
by Drew Weaver 04 Jun '21

04 Jun '21

Hello, I need to upgrade the OS that our Ceph cluster is running on to support new versions of Ceph. Has anyone devised a model for how you handle this? Do you just: Install some new nodes with the new OS Install the old version of Ceph on the new nodes Add those nodes/osds to the cluster Remove the old nodes Upgrade Ceph on the new nodes Are there any specific OS that Ceph has said that will have longer future version support? Would like to only touch the OS every 3-4 years if possible. Thanks, -Drew

4 3

CRUSH rule for EC 6+2 on 6-node cluster
by Bryan Stillwell 04 Jun '21

04 Jun '21

I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host. If I use the default rule with an osd failure domain like this: step take default step choose indep 0 type osd step emit I get clustering of 3-4 chunks on some of the hosts: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do > echo $pg > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do > ceph osd find $osd | jq -r '.host' > done | sort | uniq -c | sort -n -k1 8.0 1 harrahs 3 paris 4 aladdin 8.1 1 aladdin 1 excalibur 2 mandalaybay 4 paris 8.2 1 harrahs 2 aladdin 2 mirage 3 paris ... However, if I change the rule to use: step take default step choose indep 0 type host step chooseleaf indep 2 type osd step emit I get the data spread across 4 hosts with 2 chunks per host: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do > echo $pg > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do > ceph osd find $osd | jq -r '.host' > done | sort | uniq -c | sort -n -k1 > done 8.0 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.1 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.2 2 harrahs 2 mandalaybay 2 mirage 2 paris ... Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2. Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab). Thanks, Bryan

5 12

SAS vs SATA for OSD
by Dave Hall 04 Jun '21

04 Jun '21

Hello, We're planning another batch of OSD nodes for our cluster. Our prior nodes have been 8 x 12TB SAS drives plus 500GB NVMe per HDD. Due to market circumstances and the shortage of drives those 12TB SAS drives are in short supply. Our integrator has offered an option of 8 x 14TB SATA drives (still Enterprise). For Ceph, will the switch to SATA carry a performance difference that I should be concerned about? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

4 7

OSD Won't Start - LVM IOCTL Error - Read-only
by Dave Hall 03 Jun '21

03 Jun '21

Hello, I had an OSD drop out a couple days ago. This is 14.2.16, Bluestore, HDD + NVMe, non-container. The HDD sort of went away. I powered down the node, reseated the drive, and it came back. However, the OSD won't start. Systemctl --failed shows that the lvm2 pvscan failed, preventing the OSD unit from starting. Running the pvscan activate command manually with with verbose gave 'device-mapper: reload ioctl on (253:7) failed: Read-only file system'. I have been looking at this for a while, but I can't figure out what is read-only that is causing the problem. The full output of the pvscan is: # pvscan --cache --activate ay --verbose '8:48' pvscan devices on command line. activation/auto_activation_volume_list configuration setting not defined: All logical volumes will be auto-activated. Activating logical volume ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79/osd-block-425faf92-449e-4b57-98f2-a90a7f60e2a4. activation/volume_list configuration setting not defined: Checking only host tags for ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79/osd-block-425faf92-449e-4b57-98f2-a90a7f60e2a4. Creating ceph--block--b1fea172--71a4--463e--a3e3--8cdcc1bc7b79-osd--block--425faf92--449e--4b57--98f2--a90a7f60e2a4 Loading table for ceph--block--b1fea172--71a4--463e--a3e3--8cdcc1bc7b79-osd--block--425faf92--449e--4b57--98f2--a90a7f60e2a4 (253:7). device-mapper: reload ioctl on (253:7) failed: Read-only file system Removing ceph--block--b1fea172--71a4--463e--a3e3--8cdcc1bc7b79-osd--block--425faf92--449e--4b57--98f2--a90a7f60e2a4 (253:7) Activated 0 logical volumes in volume group ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79. 0 logical volume(s) in volume group "ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79" now active ceph-block-b1fea172-71a4-463e-a3e3-8cdcc1bc7b79: autoactivation failed. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

radosgw-admin bucket delete linear memory growth?
by Janne Johansson 03 Jun '21

03 Jun '21

I am seeing a huge usage of ram, while my bucket delete is churning over left-over multiparts, and while I realize there are *many* being done a 1000 at a time, like this: 2021-06-03 07:29:06.408 7f9b7f633240 0 abort_bucket_multiparts WARNING : aborted 254000 incomplete multipart uploads ..my first run ended radosgw-admin with out-of-mem, so it seems some part of this keeps data or forgets to free up old parts of the lists after they have been cancelled? I also don't know if restarting means it iterates over the same ones again, or if this log line means 254k are processed and if I need to restart again, it will at least have made a few hours of progress. At this point, RES is 2.1g so roughly 10000 bytes per "entry" if it is linear somehow. ceph 13.2.10 in this case. -- May the most significant bit of your life be positive.

1 0

ceph configuration using ubuntu 18.04
by Michel Niyoyita 03 Jun '21

03 Jun '21

Dear Ceph user, I want to configure ceph in our production environment using ubuntu . anyone who used it can help for the tutorial used . Best regards.

1 0

ceph-client homebrew for MacOS
by kefu chai 03 Jun '21

03 Jun '21

hi folks, i am glad to announce that with the help of Sven and Chris, ceph-client homebrew for MacOS was updated with a recent master commit. this homebrew provides some essential client side tools and libraries which allow us to talk to a ceph cluster from MacOS machine, like: ceph ceph-conf ceph-fuse rados rbd in addition to the executables and libraries, the brew also packages the header files. so, if you are up to do some development with librados, librbd and libcephfs, you can also have the necessary bits from the brew. see https://github.com/mulbc/homebrew-ceph-client it comes with a pre-bottled brew for BigSur on amd64. the last release[0] of the brew was packaged 3 years ago with mimic-13.2.2 . please note, this is not official - do not expect frequent updates or support from Ceph. cheers, -- [0] https://ceph-users.ceph.narkive.com/bnyGalkH/ceph-client-libraries-for-osx -- Regards Kefu Chai

1 0

MDS cache tunning
by Andres Rojas Guerrero 02 Jun '21

02 Jun '21

Hi all, I have observed that the MDS Cache Configuration has 18 parameters: mds_cache_memory_limit mds_cache_reservation mds_health_cache_threshold mds_cache_trim_threshold mds_cache_trim_decay_rate mds_recall_max_caps mds_recall_max_decay_threshold mds_recall_max_decay_rate mds_recall_global_max_decay_threshold mds_recall_warning_threshold mds_recall_warning_decay_rate mds_session_cap_acquisition_throttle mds_session_cap_acquisition_decay_rate mds_session_max_caps_throttle_ratio mds_cap_acquisition_throttle_retry_request_timeout mds_session_cache_liveness_magnitude mds_session_cache_liveness_decay_rate mds_max_caps_per_client I find the Ceph documentation in this section a bit cryptic and I have tried to find some resources that talk about how to tune these parameters, but without success. Does anyone have experience in adjusting these parameters according to the characteristics of the Ceph cluster itself, the hardware and the use of MDS? Regards! -- ******************************************************* Andrés Rojas Guerrero Unidad Sistemas Linux Area Arquitectura Tecnológica Secretaría General Adjunta de Informática Consejo Superior de Investigaciones Científicas (CSIC) Pinar 19 28006 - Madrid Tel: +34 915680059 -- Ext. 990059 email: a.rojas(a)csic.es ID comunicate.csic.es: @50852720l:matrix.csic.es *******************************************************

2 11

time duration of radosgw-admin
by Rok Jaklič 02 Jun '21

02 Jun '21

Hi, is it normal that radosgw-admin user info --uid=user ... takes around 3s or more? Also other radosgw-admin are taking quite a lot of time. Kind regards, Rok

2 1

Redeploy iSCSI Gateway fail - 167 returned from docker run
by Paul Giralt (pgiralt) 02 Jun '21

02 Jun '21

CEPH 16.2.4. I was having an issue where I put a server into maintenance mode and after doing so, the containers for the iSCSI gateway were not running, so I decided to do a redeploy of the service. This caused all the servers running iSCSI to get in a state where it looks like ceph orch was trying to delete the container, but it was stuck. My only recourse was to reboot the servers. I ended up doing a ‘ceph orch rm iscsi.iscsi’ to just remove the services and then tried to redeploy. When I do this, I’m seeing the following in the cephadm logs on the servers where the iscsi gateway is being deployed: 2021-06-01 19:48:15,110 INFO Deploy daemon iscsi.iscsi.cxcto-c240-j27-02.zeypah ... 2021-06-01 19:48:15,111 DEBUG Running command: /bin/docker run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 -e NODE_NAME=cxcto-c240-j27-02.cisco.com -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 -c %u %g /var/lib/ceph 2021-06-01 19:48:15,529 DEBUG stat: 167 167 Later in the logs I see: 2021-06-01 19:48:25,933 DEBUG Running command: /bin/docker inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-a67d529e-ba7f-11eb-940b-5c838f8013a5-iscsi.iscsi.cxcto-c240-j27-02.zeypah 2021-06-01 19:48:25,984 DEBUG /bin/docker: 2021-06-01 19:48:25,984 DEBUG /bin/docker: Error: No such object: ceph-a67d529e-ba7f-11eb-940b-5c838f8013a5-iscsi.iscsi.cxcto-c240-j27-02.zeypah Obviously no such object because the container creation failed. If I try to run that command that is in the logs manually, I get: [root@cxcto-c240-j27-02 ceph]# /bin/docker run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 -e NODE_NAME=cxcto-c240-j27-02.cisco.com -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 -c %u %g /var/lib/ceph stat: cannot stat '%g': No such file or directory 167 So the 167 seems to line up with what’s showing up in the script. I’m not clear on what the deal is with the %g. What is supposed to be in that placeholder? Any thoughts on why this is failing? Right now all my iSCSI gateways are down and basically my whole environment is down as a result 🙁 -Paul

1 1

Fwd: Re: Ceph osd will not start.
by Marco Pizzolo 01 Jun '21

01 Jun '21

Peter, We're seeing the same issues as you are. We have 2 new hosts Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz w/ 48 cores, 384GB RAM, and 60x 10TB SED drives and we have tried both 15.2.13 and 16.2.4 Cephadm does NOT properly deploy and activate OSDs on Ubuntu 20.04.2 with Docker. Seems to be a bug in Cephadm and a product regression, as we have 4 near identical nodes on Centos running Nautilus (240 x 10TB SED drives) and had no problems. FWIW we had no luck yet with one-by-one OSD daemon additions through ceph orch either. We also reproduced the issue easily in a virtual lab using small virtual disks on a single ceph VM with 1 mon. We are now looking into whether we can get past this with a manual buildout. If you, or anyone, has hit the same stumbling block and gotten past it, I would really appreciate some guidance. Thanks, Marco On Thu, May 27, 2021 at 2:23 PM Peter Childs <pchilds(a)bcs.org> wrote: > In the end it looks like I might be able to get the node up to about 30 > odds before it stops creating any more. > > Or more it formats the disks but freezes up starting the daemons. > > I suspect I'm missing somthing I can tune to get it working better. > > If I could see any error messages that might help, but I'm yet to spit > anything. > > Peter. > > On Wed, 26 May 2021, 10:57 Eugen Block, <eblock(a)nde.ag> wrote: > > > > If I add the osd daemons one at a time with > > > > > > ceph orch daemon add osd drywood12:/dev/sda > > > > > > It does actually work, > > > > Great! > > > > > I suspect what's happening is when my rule for creating osds run and > > > creates them all-at-once it ties the orch it overloads cephadm and it > > can't > > > cope. > > > > It's possible, I guess. > > > > > I suspect what I might need to do at least to work around the issue is > > set > > > "limit:" and bring it up until it stops working. > > > > It's worth a try, yes, although the docs state you should try to avoid > > it, it's possible that it doesn't work properly, in that case create a > > bug report. ;-) > > > > > I did work out how to get ceph-volume to nearly work manually. > > > > > > cephadm shell > > > ceph auth get client.bootstrap-osd -o > > > /var/lib/ceph/bootstrap-osd/ceph.keyring > > > ceph-volume lvm create --data /dev/sda --dmcrypt > > > > > > but given I've now got "add osd" to work, I suspect I just need to fine > > > tune my osd creation rules, so it does not try and create too many osds > > on > > > the same node at the same time. > > > > I agree, no need to do it manually if there is an automated way, > > especially if you're trying to bring up dozens of OSDs. > > > > > > Zitat von Peter Childs <pchilds(a)bcs.org>: > > > > > After a bit of messing around. I managed to get it somewhat working. > > > > > > If I add the osd daemons one at a time with > > > > > > ceph orch daemon add osd drywood12:/dev/sda > > > > > > It does actually work, > > > > > > I suspect what's happening is when my rule for creating osds run and > > > creates them all-at-once it ties the orch it overloads cephadm and it > > can't > > > cope. > > > > > > service_type: osd > > > service_name: osd.drywood-disks > > > placement: > > > host_pattern: 'drywood*' > > > spec: > > > data_devices: > > > size: "7TB:" > > > objectstore: bluestore > > > > > > I suspect what I might need to do at least to work around the issue is > > set > > > "limit:" and bring it up until it stops working. > > > > > > I did work out how to get ceph-volume to nearly work manually. > > > > > > cephadm shell > > > ceph auth get client.bootstrap-osd -o > > > /var/lib/ceph/bootstrap-osd/ceph.keyring > > > ceph-volume lvm create --data /dev/sda --dmcrypt > > > > > > but given I've now got "add osd" to work, I suspect I just need to fine > > > tune my osd creation rules, so it does not try and create too many osds > > on > > > the same node at the same time. > > > > > > > > > > > > On Wed, 26 May 2021 at 08:25, Eugen Block <eblock(a)nde.ag> wrote: > > > > > >> Hi, > > >> > > >> I believe your current issue is due to a missing keyring for > > >> client.bootstrap-osd on the OSD node. But even after fixing that > > >> you'll probably still won't be able to deploy an OSD manually with > > >> ceph-volume because 'ceph-volume activate' is not supported with > > >> cephadm [1]. I just tried that in a virtual environment, it fails when > > >> activating the systemd-unit: > > >> > > >> ---snip--- > > >> [2021-05-26 06:47:16,677][ceph_volume.process][INFO ] Running > > >> command: /usr/bin/systemctl enable > > >> ceph-volume@lvm-8-1a8fc8ae-8f4c-4f91-b044-d5636bb52456 > > >> [2021-05-26 06:47:16,692][ceph_volume.process][INFO ] stderr Failed > > >> to connect to bus: No such file or directory > > >> [2021-05-26 06:47:16,693][ceph_volume.devices.lvm.create][ERROR ] lvm > > >> activate was unable to complete, while creating the OSD > > >> Traceback (most recent call last): > > >> File > > >> "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", > > >> line 32, in create > > >> Activate([]).activate(args) > > >> File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", > > >> line 16, in is_root > > >> return func(*a, **kw) > > >> File > > >> > "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py", > > >> line > > >> 294, in activate > > >> activate_bluestore(lvs, args.no_systemd) > > >> File > > >> > "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py", > > >> line > > >> 214, in activate_bluestore > > >> systemctl.enable_volume(osd_id, osd_fsid, 'lvm') > > >> File > > >> "/usr/lib/python3.6/site-packages/ceph_volume/systemd/systemctl.py", > > >> line 82, in enable_volume > > >> return enable(volume_unit % (device_type, id_, fsid)) > > >> File > > >> "/usr/lib/python3.6/site-packages/ceph_volume/systemd/systemctl.py", > > >> line 22, in enable > > >> process.run(['systemctl', 'enable', unit]) > > >> File "/usr/lib/python3.6/site-packages/ceph_volume/process.py", > > >> line 153, in run > > >> raise RuntimeError(msg) > > >> RuntimeError: command returned non-zero exit status: 1 > > >> [2021-05-26 06:47:16,694][ceph_volume.devices.lvm.create][INFO ] will > > >> rollback OSD ID creation > > >> [2021-05-26 06:47:16,697][ceph_volume.process][INFO ] Running > > >> command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > > >> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.8 > > >> --yes-i-really-mean-it > > >> [2021-05-26 06:47:17,597][ceph_volume.process][INFO ] stderr purged > > osd.8 > > >> ---snip--- > > >> > > >> There's a workaround described in [2] that's not really an option for > > >> dozens of OSDs. I think your best approach is to bring cephadm to > > >> activate the OSDs for you. > > >> You wrote you didn't find any helpful error messages, but did cephadm > > >> even try to deploy OSDs? What does your osd spec file look like? Did > > >> you explicitly run 'ceph orch apply osd -i specfile.yml'? This should > > >> trigger cephadm and you should see at least some output like this: > > >> > > >> Mai 26 08:21:48 pacific1 conmon[31446]: 2021-05-26T06:21:48.466+0000 > > >> 7effc15ff700 0 log_channel(cephadm) log [INF] : Applying service > > >> osd.ssd-hdd-mix on host pacific2... > > >> Mai 26 08:21:49 pacific1 conmon[31009]: cephadm > > >> 2021-05-26T06:21:48.469611+0000 mgr.pacific1.whndiw (mgr.14166) 1646 : > > >> cephadm [INF] Applying service osd.ssd-hdd-mix on host pacific2... > > >> > > >> Regards, > > >> Eugen > > >> > > >> [1] https://tracker.ceph.com/issues/49159 > > >> [2] https://tracker.ceph.com/issues/46691 > > >> > > >> > > >> Zitat von Peter Childs <pchilds(a)bcs.org>: > > >> > > >> > Not sure what I'm doing wrong, I suspect its the way I'm running > > >> > ceph-volume. > > >> > > > >> > root@drywood12:~# cephadm ceph-volume lvm create --data /dev/sda > > >> --dmcrypt > > >> > Inferring fsid 1518c8e0-bbe4-11eb-9772-001e67dc85ea > > >> > Using recent ceph image ceph/ceph@sha256 > > >> > :54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > > >> > /usr/bin/docker: Running command: /usr/bin/ceph-authtool > > --gen-print-key > > >> > /usr/bin/docker: Running command: /usr/bin/ceph-authtool > > --gen-print-key > > >> > /usr/bin/docker: --> RuntimeError: No valid ceph configuration file > > was > > >> > loaded. > > >> > Traceback (most recent call last): > > >> > File "/usr/sbin/cephadm", line 8029, in <module> > > >> > main() > > >> > File "/usr/sbin/cephadm", line 8017, in main > > >> > r = ctx.func(ctx) > > >> > File "/usr/sbin/cephadm", line 1678, in _infer_fsid > > >> > return func(ctx) > > >> > File "/usr/sbin/cephadm", line 1738, in _infer_image > > >> > return func(ctx) > > >> > File "/usr/sbin/cephadm", line 4514, in command_ceph_volume > > >> > out, err, code = call_throws(ctx, c.run_cmd(), > > verbosity=verbosity) > > >> > File "/usr/sbin/cephadm", line 1464, in call_throws > > >> > raise RuntimeError('Failed command: %s' % ' '.join(command)) > > >> > RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host > > >> > --net=host --entrypoint /usr/sbin/ceph-volume --privileged > > >> --group-add=disk > > >> > --init -e CONTAINER_IMAGE=ceph/ceph@sha256 > :54e95ae1e11404157d7b329d0t > > >> > > > >> > root@drywood12:~# cephadm shell > > >> > Inferring fsid 1518c8e0-bbe4-11eb-9772-001e67dc85ea > > >> > Inferring config > > >> > > > /var/lib/ceph/1518c8e0-bbe4-11eb-9772-001e67dc85ea/mon.drywood12/config > > >> > Using recent ceph image ceph/ceph@sha256 > > >> > :54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > > >> > root@drywood12:/# ceph-volume lvm create --data /dev/sda --dmcrypt > > >> > Running command: /usr/bin/ceph-authtool --gen-print-key > > >> > Running command: /usr/bin/ceph-authtool --gen-print-key > > >> > Running command: /usr/bin/ceph --cluster ceph --name > > client.bootstrap-osd > > >> > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > > >> > 70054a5c-c176-463a-a0ac-b44c5db0987c > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable > to > > >> find > > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such > > file > > >> or > > >> > directory > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 > > >> > AuthRegistry(0x7fdef405b378) no keyring found at > > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable > to > > >> find > > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such > > file > > >> or > > >> > directory > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 > > >> > AuthRegistry(0x7fdef405ef20) no keyring found at > > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable > to > > >> find > > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such > > file > > >> or > > >> > directory > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 > > >> > AuthRegistry(0x7fdef8f0bea0) no keyring found at > > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef2d9d700 -1 > > monclient(hunting): > > >> > handle_auth_bad_method server allowed_methods [2] but i only support > > [1] > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef259c700 -1 > > monclient(hunting): > > >> > handle_auth_bad_method server allowed_methods [2] but i only support > > [1] > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef1d9b700 -1 > > monclient(hunting): > > >> > handle_auth_bad_method server allowed_methods [2] but i only support > > [1] > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 monclient: > > >> > authenticate NOTE: no keyring found; disabled cephx authentication > > >> > stderr: [errno 13] RADOS permission denied (error connecting to the > > >> > cluster) > > >> > --> RuntimeError: Unable to create a new OSD id > > >> > root@drywood12:/# lsblk /dev/sda > > >> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > >> > sda 8:0 0 7.3T 0 disk > > >> > > > >> > As far as I can see cephadm gets a little further than this as the > > disks > > >> > have lvm volumes on them just the osd's daemons are not created or > > >> started. > > >> > So maybe I'm invoking ceph-volume incorrectly. > > >> > > > >> > > > >> > On Tue, 25 May 2021 at 06:57, Peter Childs <pchilds(a)bcs.org> wrote: > > >> > > > >> >> > > >> >> > > >> >> On Mon, 24 May 2021, 21:08 Marc, <Marc(a)f1-outsourcing.eu> wrote: > > >> >> > > >> >>> > > > >> >>> > I'm attempting to use cephadm and Pacific, currently on debian > > >> buster, > > >> >>> > mostly because centos7 ain't supported any more and cenotos8 > ain't > > >> >>> > support > > >> >>> > by some of my hardware. > > >> >>> > > >> >>> Who says centos7 is not supported any more? Afaik centos7/el7 is > > being > > >> >>> supported till its EOL 2024. By then maybe a good alternative for > > >> >>> el8/stream has surfaced. > > >> >>> > > >> >> > > >> >> Not supported by ceph Pacific, it's our os of choice otherwise. > > >> >> > > >> >> My testing says the version available of podman, docker and > python3, > > do > > >> >> not work with Pacific. > > >> >> > > >> >> Given I've needed to upgrade docker on buster can we please have a > > list > > >> of > > >> >> versions that work with cephadm, maybe even have cephadm say no, > > please > > >> >> upgrade unless your running the right version or better. > > >> >> > > >> >> > > >> >> > > >> >>> > Anyway I have a few nodes with 59x 7.2TB disks but for some > reason > > >> the > > >> >>> > osd > > >> >>> > daemons don't start, the disks get formatted and the osd are > > created > > >> but > > >> >>> > the daemons never come up. > > >> >>> > > >> >>> what if you try with > > >> >>> ceph-volume lvm create --data /dev/sdi --dmcrypt ? > > >> >>> > > >> >> > > >> >> I'll have a go. > > >> >> > > >> >> > > >> >>> > They are probably the wrong spec for ceph (48gb of memory and > > only 4 > > >> >>> > cores) > > >> >>> > > >> >>> You can always start with just configuring a few disks per node. > > That > > >> >>> should always work. > > >> >>> > > >> >> > > >> >> That was my thought too. > > >> >> > > >> >> Thanks > > >> >> > > >> >> Peter > > >> >> > > >> >> > > >> >>> > but I was expecting them to start and be either dirt slow or > crash > > >> >>> > later, > > >> >>> > anyway I've got upto 30 of them, so I was hoping on getting at > > least > > >> get > > >> >>> > 6PB of raw storage out of them. > > >> >>> > > > >> >>> > As yet I've not spotted any helpful error messages. > > >> >>> > > > >> >>> _______________________________________________ > > >> >>> ceph-users mailing list -- ceph-users(a)ceph.io > > >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> >>> > > >> >> > > >> > _______________________________________________ > > >> > ceph-users mailing list -- ceph-users(a)ceph.io > > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> > > >> > > >> _______________________________________________ > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 11

cephadm removed mon. key when adding new mon node
by Bryan Stillwell 01 Jun '21

01 Jun '21

This morning I tried adding a mon node to my home Ceph cluster with the following command: ceph orch daemon add mon ether This seemed to work at first, but then it decided to remove it fairly quickly which broke the cluster because the mon. keyring was also removed: 2021-06-01T14:16:11.523210+0000 mgr.paris.glbvov [INF] Deploying daemon mon.ether on ether 2021-06-01T14:16:43.621759+0000 mgr.paris.glbvov [INF] Safe to remove mon.ether: not in monmap (['paris', 'excalibur']) 2021-06-01T14:16:43.622135+0000 mgr.paris.glbvov [INF] Removing monitor ether from monmap... 2021-06-01T14:16:43.641365+0000 mgr.paris.glbvov [INF] Removing daemon mon.ether from ether 2021-06-01T14:16:46.610283+0000 mgr.paris.glbvov [INF] Removing key for mon. Digging in to this it seems like this line might need to check for 'mon.' and not 'mon': https://github.com/ceph/ceph/blob/master/src/pybind/mgr/cephadm/services/ce… Anyways, does anyone know how to import the mon. keyring again once it has been removed? Thanks, Bryan

1 1

CentOS 7 dependencies for diskprediction module
by Michal Strnad 01 Jun '21

01 Jun '21

Hi, Did anyone get the diskprediction-local plugin working on CentOS 7.9? When we enable the plugin with v14.2.19 we get following error. Module 'diskprediction_local' has failed: No module named sklearn.svm.classes If the package is installed it brings in several dependencies but apparently these are not enough. Installing: ceph-mgr-diskprediction-local noarch 2:14.2.19-0.el7 Ceph-noarch 1.1 M Installing for dependencies: atlas x86_64 3.10.1-12.el7 base 4.5 M blas x86_64 3.4.2-8.el7 base 399 k lapack x86_64 3.4.2-8.el7 base 5.4 M libgfortran x86_64 4.8.5-44.el7 base 301 k libquadmath x86_64 4.8.5-44.el7 base 190 k numpy x86_64 1:1.7.1-13.el7 base 2.8 M numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k python-devel x86_64 2.7.5-90.el7 updates 399 k python-nose noarch 1.3.7-1.el7 base 276 k python2-rpm-macros noarch 3-34.el7 base 8.1 k scipy x86_64 0.12.1-6.el7 base 9.3 M suitesparse x86_64 4.0.2-10.el7 base 928 k tbb x86_64 4.1-9.20130314.el7 base 124 k We tried using pip/pip3 to get another library but diskprediction_local still doesn't work. Are we missing something? Thank you Cheers, Michal

1 0

Unable to delete disk from iSCSI target
by Paul Giralt (pgiralt) 01 Jun '21

01 Jun '21

I’m trying to delete a disk from an iSCSI target so that I can remove the image, but running into an issue. If I try to delete it from the CEPH dashboard, I just get an error saying that the DELETE timed out after 45 seconds. If I try to do it from gwcli, the command never returns: /iscsi-target...4075058/disks> delete iscsi-pool-001/iscsi-img-0003 After I enter that command, it just hangs indefinitely. I’ve tried to remove any dependencies on the disk by making sure it’s unmounted and detached from the ESXi hosts that are using the disk. I’m guessing it’s failing because there is some dependency on the disk that is causing the deletion to fail, but not sure what else to look at. Any thoughts? -Paul

1 0

Cephfs metadta pool suddenly full (100%) !
by Hervé Ballans 01 Jun '21

01 Jun '21

Hi all, Ceph Nautilus 14.2.16. We encounter a strange and critical poblem since this morning. Our cephfs metadata pool suddenly grew from 2,7% to 100%! (in less than 5 hours) while there is no significant activities on the OSD data ! Here are some numbers: # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 205 TiB 103 TiB 102 TiB 102 TiB 49.68 nvme 4.4 TiB 2.2 TiB 2.1 TiB 2.2 TiB 49.63 TOTAL 210 TiB 105 TiB 104 TiB 104 TiB 49.68 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_data_home 7 512 11 TiB 22.58M 11 TiB 18.31 17 TiB cephfs_metadata_home 8 128 724 GiB 2.32M 724 GiB 100.00 0 B rbd_backup_vms 9 1024 19 TiB 5.00M 19 TiB 37.08 11 TiB The cephfs_data uses less than the half of the storage space, and there was no significant increase during the period (and before) where metadata became full. Is someone already encounter that ? Currently, I have no idea how I can solve this problem. The restart of associated OSD and mds services have not been useful. Let me know if you want more informations or logs. Thank you for your help. Regards, Hervé

2 3

HEALTH_WARN and osd zero size
by julien lenseigne 01 Jun '21

01 Jun '21

Dear all, I have a problem with a ceph cluster. reconstruction is stopped. The status remains at HEALTH_WARN. Strangely enough, when I do a 'ceph osd df' I have have some OSD which returns zero in size. Maybe, is it related to my reconstruction problem. ceph status : https://pastebin.com/rEwG0nD5 ceph osd df : https://pastebin.com/P2Dn1z7A ceph osd tree : https://pastebin.com/rpy7hkiw Thanks. -- Julien Lenseigne Responsable informatique LMD Tel: 0169335172 Ecole Polytechnique, route de saclay, 91128 Palaiseau Bat 83 - Bureau 83.30.13

1 0

Re: Cephfs metadta pool suddenly full (100%) !
by Sebastian Knust 01 Jun '21

01 Jun '21

Hi Hervé, On 01.06.21 14:00, Hervé Ballans wrote: > I'm aware with your points, and maybe I was not really clear in my > previous email (written in a hurry!) > The problematic pool is the metadata one. All its OSDs (x3) are full. > The associated data pool is OK and no OSD is full on the data pool. Are you saying that you only have 3 OSD for your metadata pool, which are the full ones? Alright, then you can - at least for this specific issue - disregard my previous comment. > > The problem is that metadata pool suddenly increases a lot and > continiously from 3% to 100% in 5 hours (from 5 am to 10 am, then crash) 724 GiB stored in the metadata pool with only 11 TiB cephfs data size does seem huge at first glance. For reference, I have about 160 TiB cephfs data with only 31 GiB stored in the metadata pool. I don't have an explanation for this behaviour, as I am relatively new to Ceph. Maybe the list can chime in? > > And we don't understand the reason, since there was no specific > activities on the data pool ? > This cluster runs perfectly with the current configuration since many > years. Probably unrelated to your issues: I noticed that your STORED and USED column in `ceph df` output are identical. Is that because of Nautilus (I myself am running Octopus, where USED is the expected multiple of STORED depending on replication factor / EC configuration in the pool) or are you running a specific configuration that might cause that? Cheers Sebastian

1 0

Re: Cephfs metadta pool suddenly full (100%) !
by Sebastian Knust 01 Jun '21

01 Jun '21

Hi Hervé, On 01.06.21 13:15, Hervé Ballans wrote: > # ceph status > cluster: > id: 838506b7-e0c6-4022-9e17-2d1cf9458be6 > health: HEALTH_ERR > 1 filesystem is degraded > 3 full osd(s) > 1 pool(s) full > 1 daemons have recently crashed You have full OSDs and therefore a full pool. The "fullness" of a pool is limited by the fullest OSD, i.e. a single full OSDs can block your pool. Take a look at `ceph osd df` and you will notice a very non-uniform osd usage (both with numbers of PG / size as well as usage %). > osd: 126 osds: 126 up (since 5m), 126 in (since 5M) > pgs: 1662 active+clean The PG/osd ratio seems to be very low for me. The general recommendation is 100 PG / osd post-replication (and power of 2 for each pool). In my cluster I actually run with ~200 PG / osd for my SSD which contain the cephfs metadata. > > Thanks a lot if you have some ways for trying to solve this... You have to get your OSDs to rebalance, which probably includes increasing the number of PGs in some pools. Details depend on which Ceph version you are running and your CRUSH rules (maybe your cephfs metadata pool is residing only on NVMe?). Take a look at the balancer module [1] and the autoscaler [2] (`ceph osd pool autoscale-status` is most interesting). Theoretically, you could (temporarilly!) increase the full_ratio. However, this is a very dangerous operation which you should not do unless you know *exactly* what you are doing. Cheers & Best of luck Sebastian [1] https://docs.ceph.com/en/latest/rados/operations/balancer/ [2] https://docs.ceph.com/en/latest/rados/operations/placement-groups/ Replace latest in the URIs with your Ceph version string (i.e. octopus, nautilus) for version specific documentation

2 1

local mirror from quay.ceph.io
by Seba chanel 01 Jun '21

01 Jun '21

hi, Can you show me how to mirror the ceph docker images that are available from quay.ceph.io Our CEPH clusters are on a private network, and they are installed from a local repository because they cannot directly access the internet network. With Octopus, I need to be able to create a local mirror of the different packages and images. What do you recommend as a product for creating a local registry? thanks for your advices Sébastien.

1 0

The always welcomed large omap
by Szabo, Istvan (Agoda) 01 Jun '21

01 Jun '21

Hi, Any way to clean up large-omap in the index pool? PG deep_scrub didn't help. I know how to clean in the log pool, but no idea in the index pool :/ It's an octopus deployment 15.2.10. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Re: The always welcomed large omap
by Szabo, Istvan (Agoda) 01 Jun '21

01 Jun '21

So the bucket has been deleted on the master zone which has been removed from the other zones as well. On the master zone after deep scrub the omap disappeared but on the secondary zone it's still there. It was 3 at the beginning after I scrubbed the affected OSDs (not just pgs) I have 6 omap. The bucket which I assumed caused the issue is not there anymore. This is the log: 2021-06-01T08:48:11.560449+0700 osd.39 (osd.39) 1 : cluster [DBG] 20.6 deep-scrub starts 2021-06-01T08:48:13.218814+0700 osd.39 (osd.39) 2 : cluster [WRN] Large omap object found. Object: 20:62d24dc9:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.5:head PG: 20.93b24b46 (20.6) Key count: 244664 Size (bytes): 69659918 2021-06-01T08:48:15.245975+0700 osd.39 (osd.39) 3 : cluster [DBG] 20.6 deep-scrub ok 2021-06-01T08:48:16.623097+0700 osd.39 (osd.39) 4 : cluster [DBG] 20.e deep-scrub starts 2021-06-01T08:48:20.201926+0700 osd.39 (osd.39) 5 : cluster [WRN] Large omap object found. Object: 20:77cbb881:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.9:head PG: 20.811dd3ee (20.e) Key count: 244363 Size (bytes): 69582418 2021-06-01T08:48:20.275906+0700 osd.39 (osd.39) 6 : cluster [DBG] 20.e deep-scrub ok 2021-06-01T08:48:21.560212+0700 osd.39 (osd.39) 7 : cluster [DBG] 20.15 deep-scrub starts 2021-06-01T08:48:22.456133+0700 osd.39 (osd.39) 8 : cluster [WRN] Large omap object found. Object: 20:a8a0840e:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.3:head PG: 20.70210515 (20.15) Key count: 244169 Size (bytes): 69508179 2021-06-01T08:48:25.202051+0700 osd.39 (osd.39) 9 : cluster [DBG] 20.15 deep-scrub ok 2021-06-01T08:48:26.019422+0700 osd.37 (osd.37) 4 : cluster [DBG] 20.f deep-scrub starts 2021-06-01T08:48:28.370919+0700 osd.37 (osd.37) 5 : cluster [WRN] Large omap object found. Object: 20:f59fa8d7:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.6:head PG: 20.eb15f9af (20.f) Key count: 244243 Size (bytes): 69539068 2021-06-01T08:48:29.010877+0700 osd.37 (osd.37) 6 : cluster [DBG] 20.f deep-scrub ok 2021-06-01T08:48:29.573160+0700 osd.39 (osd.39) 10 : cluster [DBG] 20.18 deep-scrub starts 2021-06-01T08:48:32.682416+0700 osd.39 (osd.39) 11 : cluster [WRN] Large omap object found. Object: 20:1e2579ff:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.4:head PG: 20.ff9ea478 (20.18) Key count: 243858 Size (bytes): 69426645 2021-06-01T08:48:33.383843+0700 osd.39 (osd.39) 12 : cluster [DBG] 20.18 deep-scrub ok 2021-06-01T08:48:35.021940+0700 osd.37 (osd.37) 7 : cluster [DBG] 20.14 deep-scrub starts 2021-06-01T08:48:36.139892+0700 osd.37 (osd.37) 8 : cluster [WRN] Large omap object found. Object: 20:291b1669:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.2.1:head PG: 20.9668d894 (20.14) Key count: 244533 Size (bytes): 69611691 2021-06-01T08:48:38.843235+0700 osd.37 (osd.37) 9 : cluster [DBG] 20.14 deep-scrub ok Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com --------------------------------------------------- -----Original Message----- From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Sent: Monday, May 31, 2021 10:53 PM To: Matt Vandermeulen <storage(a)reenigne.net> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: [Suspicious newsletter] Re: The always welcomed large omap Yeah, I found a bucket at the moment in progress the deleting, will preshard it. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com --------------------------------------------------- -----Original Message----- From: Matt Vandermeulen <storage(a)reenigne.net> Sent: Monday, May 31, 2021 9:43 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Cc: ceph-users(a)ceph.io Subject: [Suspicious newsletter] [ceph-users] Re: The always welcomed large omap All the index data will be in OMAP, which you can see a listing of with `ceph osd df tree` Do you have large buckets (many, many objects in a single bucket) with few shards? You may have to reshard one (or some) of your buckets. It'll take some reading if you're using multisite, in order to coordinate it (though I'm unfamiliar with how it works with multisite in Octopus). On 2021-05-31 02:25, Szabo, Istvan (Agoda) wrote: > Hi, > > Any way to clean up large-omap in the index pool? > PG deep_scrub didn't help. > I know how to clean in the log pool, but no idea in the index pool :/ > It's an octopus deployment 15.2.10. > > Thank you > > ________________________________ > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by > copyright or other legal rules. If you have received it by mistake > please let us know by reply email and delete it from your system. It > is prohibited to copy this message or disclose its content to anyone. > Any confidentiality or privilege is not waived or lost by any mistaken > delivery or unauthorized disclosure of the message. All messages sent > to and from Agoda may be monitored to ensure compliance with company > policies, to protect the company's interests and to remove potential > malware. Electronic messages may be intercepted, amended, lost or > deleted, or contain viruses. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0