- ceph-users - lists.ceph.io

Weird ceph use case, is there any unknown bucket limitation?
by Szabo, Istvan (Agoda) 19 Nov '20

19 Nov '20

Hi, I have a use case where the user would like to have 50000 Buckets. Is it normal for ceph just too much for me? The reason they want this level of granularity is because they might need to clean buckets for a specific subset and not affect others. Bucket format is this: PR_PAGETPYE-<PAGE_TYPE_ID>_DEVICE-<DEVICE_ID>_WHITELABEL-<WHITELABEL_ID>_LANGUAGE-<LANGUAGE_ID>_SUBTYPE-<SUBTYPE_ID>/<OBJECT_ID> Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

MGR restart loop
by Frank Schilder 19 Nov '20

19 Nov '20

Dear cephers, I have a problem with MGR daemons, ceph version mimic-13.2.8. I'm trying to do maintenance on our MON/MGR servers and am through with 2 out of 3. I have MON and MGR collocated on a host, 3 hosts in total. So far, procedure was to stop the deamons on the server and do the maintenance. Now I'd stuck at the last server, because MGR fail-over does not work. The remaining MGR instances go into a restart loop. In an attempt to mitigate this, I stopped all but 1 MGR on a node that is done with maintenance. Everything fine. However, as soon as I stop the last MON I need to do maintenance on, the last remaining MGR goes into a restart loop all by itself. As far as I can see, the MGR does actually not restart, it just gets thrown out of the cluster. Here is a ceph status before stopping mon.ceph-01: [root@ceph-01 ~]# ceph status cluster: id: xxx health: HEALTH_WARN 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-03(active) mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 302 osds: 281 up, 281 in data: pools: 11 pools, 3215 pgs objects: 334.1 M objects, 689 TiB usage: 877 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 3208 active+clean 7 active+clean+scrubbing+deep As soon as I stop mon.ceph-01, all hell breaks loose. Note that mgr.ceph-03 is collocated with mon.ceph-03 and we have quorum between mon.ceph-02 and mon.ceph-03. Here ceph status snapshots after shutting down mon.ceph-01: [root@ceph-01 ~]# ceph status cluster: id: xxx health: HEALTH_WARN 1 pools nearfull 1/3 mons down, quorum ceph-02,ceph-03 services: mon: 3 daemons, quorum ceph-02,ceph-03, out of quorum: ceph-01 mgr: ceph-03(active) mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 302 osds: 281 up, 281 in data: pools: 11 pools, 3215 pgs objects: 334.1 M objects, 689 TiB usage: 877 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 3207 active+clean 8 active+clean+scrubbing+deep [root@ceph-01 ~]# ceph status cluster: id: xxx health: HEALTH_WARN no active mgr 1 pools nearfull 1/3 mons down, quorum ceph-02,ceph-03 services: mon: 3 daemons, quorum ceph-02,ceph-03, out of quorum: ceph-01 mgr: no daemons active mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 302 osds: 281 up, 281 in data: pools: 11 pools, 3215 pgs objects: 334.1 M objects, 689 TiB usage: 877 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 3207 active+clean 8 active+clean+scrubbing+deep [root@ceph-01 ~]# ceph status cluster: id: xxx health: HEALTH_WARN 1 pools nearfull 1/3 mons down, quorum ceph-02,ceph-03 services: mon: 3 daemons, quorum ceph-02,ceph-03, out of quorum: ceph-01 mgr: ceph-03(active, starting) mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 302 osds: 281 up, 281 in data: pools: 11 pools, 3215 pgs objects: 334.1 M objects, 689 TiB usage: 877 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 3207 active+clean 8 active+clean+scrubbing+deep It is cycling through these 3 states and I couldn't find a reason why. The node ceph-01 is not special in any way. Any hint would be greatly appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 2

v14.2.14 Nautilus released
by David Galloway 18 Nov '20

18 Nov '20

This is the 14th backport release in the Nautilus series. This releases fixes a security flaw affecting Messenger V2 for Octopus & Nautilus, among other fixes across components. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-14-nautilus-released Notable Changes --------------- * CVE 2020-25660: Fix a regression in Messenger V2 replay attacks Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.14.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7e94c5afc28f3eaf36151ad1e1457de5f16c4fdf

1 0

CentOS 8, Ceph Octopus, ssh private key
by Mika Saari 18 Nov '20

18 Nov '20

Hi, Installing Ceph Octopus using cephadm. I managed to install ceph-common with cephadm and when trying to add new hosts with "ceph orch host add ceph2" I get error "Error EINVAL: Failed to connect to ceph2 (ceph2). Check that the host is reachable and accepts connection using the cephadm SSH key". I verified that I am able to ssh login to the ceph2 server with ceph private_key like it was described in the error message. But since adding new hosts to the ceph wasn't working, I tried generating the new private key and updating the public keys to the remote servers with: # ceph cephadm clear-key # ceph cephadm generate-key # ceph cephadm get-pub-key > ceph.pub # ceph config-key get mgr/cephadm/ssh_identity_key > ceph.priv # ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph1 # ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2 # ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3 And then testing that the private key really is working: # chmod 600 ceph.priv # ssh -i ceph.priv root@ceph2 At this point ssh works with passwordless login. But still the ceph orch host add ceph2 doesn't work (giving exactly the same error) I also tried restarting the manager with "ceph mgr fail" which was informed somewhere -> no effect. Also tried rebooting the machines -> no effect. Any tips I could still try ? Thank you very much!

2 3

Using rbd-nbd tool in Ceph development cluster
by Bobby 18 Nov '20

18 Nov '20

Hi, I have to use this *rbd-nbd *tool from Ceph. This is part of Ceph source code. Here: https://github.com/ceph/ceph/tree/master/src/tools/rbd_nbd My question is: Can we use this *rbd-nbd* tool in the Ceph cluster? By Ceph cluster I mean the development cluster we build through *vstart.sh* script. I am quite sure we could use it. I have this script running. I can *start* and *stop* the cluster. But I am struggling to use this rbd-nbd tool in the development cluster which we build through vstart.sh script. Looking for help. Thanks.

2 2

MONs unresponsive for excessive amount of time
by Frank Schilder 18 Nov '20

18 Nov '20

Hi all, one of our MONs was down for maintenance for ca. 45 minutes. After this time I started it up again and it joined the cluster. Unfortunately, things did not go as expected. The MON sub-cluster became unresponsive for a bit more than 10 minutes. Admin commands would hang, even if issued directly to a specific monitor via "ceph tell mon.xxx". In addition, our MDS lost connection to the MONs and reported a laggy connection. Consequently, all ceph fs access was frozen for a bit more than 10 minutes as well. From the little I could get out with "ceph daemon mon.xxx mon_status" I could see that the restarted MON was in state "synchronizing" (or similar, its from memory) while the other mons were in quorum. Our cluster is mimic-12.2.8. Somehow, this observation does not fit together with the intended HA of the MON cluster, there should not be any stall at all. My questions: Why do the MONs become unresponsive for such a long time? What are the MONs doing during this time frame? Are there any config options I should look at? Are there any log messages I should hunt for? Any hint is appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Not all OSDs in rack marked as down when the rack fails
by Wido den Hollander 18 Nov '20

18 Nov '20

Hi, I'm investigating an issue where 4 to 5 OSDs in a rack aren't marked as down when the network is cut to that rack. Situation: - Nautilus cluster - 3 racks - 120 OSDs, 40 per rack We performed a test where we turned off the network Top-of-Rack for each rack. This worked as expected with two racks, but with the third something weird happened. From the 40 OSDs which were supposed to be marked as down only 36 were marked as down. In the end it took 15 minutes for all 40 OSDs to be marked as down. $ ceph config set mon mon_osd_reporter_subtree_level rack That setting is set to make sure that we only accept reports from other racks. What we saw in the logs for example: 2020-10-29T03:49:44.409-0400 7fbda185e700 10 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 osd.51 has 54 reporters, 239.856038 grace (20.000000 + 219.856 + 7.43801e-23), max_failed_since 2020-10-29T03:47:22.374857-0400 But osd.51 was still not marked as down after 54 reporters have reported that it is actually down. I checked, no ping or other traffic possible to osd.51. Host is unreachable. Another osd was marked as down, but it took a couple of minutes as well: 2020-10-29T03:50:54.455-0400 7fbda185e700 10 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 osd.37 has 48 reporters, 221.378970 grace (20.000000 + 201.379 + 6.34437e-23), max_failed_since 2020-10-29T03:47:12.761584-0400 2020-10-29T03:50:54.455-0400 7fbda185e700 1 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 we have enough reporters to mark osd.37 down In the end osd.51 was marked as down, but only after the MON decided to do so: 2020-10-29T03:53:44.631-0400 7fbda185e700 0 log_channel(cluster) log [INF] : osd.51 marked down after no beacon for 903.943390 seconds 2020-10-29T03:53:44.631-0400 7fbda185e700 -1 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107104 no beacon from osd.51 since 2020-10-29T03:38:40.689062-0400, 903.943390 seconds ago. marking down I haven't seen this happen before in any cluster. It's also strange that this only happens in this rack, the other two racks work fine. ID CLASS WEIGHT TYPE NAME -1 1545.35999 root default -206 515.12000 rack 206 -7 27.94499 host CEPH2-206-U16 ... -207 515.12000 rack 207 -17 27.94499 host CEPH2-207-U16 ... -208 515.12000 rack 208 -31 27.94499 host CEPH2-208-U16 ... That's how the CRUSHMap looks like. Straight forward and 3x replication over 3 racks. This issue only occurs in rack *207*. Has anybody seen this before or knows where to start? Wido

2 3

Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type'
by Marcelo 18 Nov '20

18 Nov '20

Hello all. I'm trying to deploy the dashboard (Nautilus 14.2.8), and after I run ceph dashboard create-self-signed-cert, the cluster started to show this warning: # ceph health detail HEALTH_ERR Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type' MGR_MODULE_ERROR Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type' Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type' If I set ceph config set mgr mgr/dashboard/ssl false, the error goes away. I tried to manually upload the certs, but I'm still hitting the error. Has anyone experienced something similar? Thanks, Marcelo.

2 2

Unable to clarify error using vfs_ceph (Samba gateway for CephFS)
by Matt Larson 18 Nov '20

18 Nov '20

I am getting an error in the log.smbd from the Samba gateway that I don’t understand and looking for help from anyone who has gotten the vfs_ceph working. Background: I am trying to get a Samba gateway with CephFS working with the vfs_ceph module. I observed that the default Samba package on CentOS 7.7 did not come with the ceph.so vfs_ceph module, so I tried to compile a working Samba version with vfs_ceph. Newer Samba versions have a requirement for GnuTLS >= 3.4.7, which is not an available package on CentOS 7.7 without a custom repository. I opted to build an earlier version of Samba. On CentOS 7.7, I built Samba 4.11.16 with [global] security = user map to guest = Bad User username map = /etc/samba/smbusers log level = 4 load printers = no printing = bsd printcap name = /dev/null disable spoolss = yes [cryofs_upload] public = yes read only = yes guest ok = yes vfs objects = ceph path = /upload kernel share modes = no ceph:user_id = samba.upload ceph:config_file = /etc/ceph/ceph.conf I have a file at /etc/ceph/ceph.conf including: fsid = redacted mon_host = redacted auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx I have an /etc/ceph/client.samba.upload.keyring /w key for the user `samba.upload` However, connecting fails: smbclient \\\\localhost\\cryofs_upload -U guest Enter guest's password: tree connect failed: NT_STATUS_UNSUCCESSFUL The log.smbd gives these errors: Initialising custom vfs hooks from [ceph] [2020/11/11 17:24:37.388460, 3] ../../lib/util/modules.c:167(load_module_absolute_path) load_module_absolute_path: Module '/usr/local/samba/lib/vfs/ceph.so' loaded [2020/11/11 17:24:37.402026, 1] ../../source3/smbd/service.c:668(make_connection_snum) make_connection_snum: SMB_VFS_CONNECT for service 'cryofs_upload' at '/upload' failed: No such file or directory There is an /upload directory for which the samba.upload user has read access to in the CephFS. What does this error mean: ‘no such file or directory’ ? Is it that vfs_ceph isn’t finding `/upload` or is some other file depended by vfs_ceph not been found? I have also tried to specify a local path rather than a CephFS path and will get the same error. Is there any good guide that describes not just the Samba smb.conf, but also what should be in /etc/ceph/ceph.conf, and how to provide the key for the ceph:user_id ? I am really struggling to find good first-hand documentation for this. Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

4 5

Ceph EC PG calculation
by Szabo, Istvan (Agoda) 18 Nov '20

18 Nov '20

Hi, I have this error: I have 36 osd and get this: Error ERANGE: pg_num 4096 size 6 would mean 25011 total pgs, which exceeds max 10500 (mon_max_pg_per_osd 250 * num_in_osds 42) If I want to calculate the max pg in my server, how it works if I have EC pool? I have 4:2 data EC pool, and the others are replicated. These are the pools: pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode warn last_change 597 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 598 flags hashpspool stripe_width 0 application rgw pool 6 'sin.rgw.log' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 599 flags hashpspool stripe_width 0 application rgw pool 7 'sin.rgw.control' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 600 flags hashpspool stripe_width 0 application rgw pool 8 'sin.rgw.meta' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 601 lfor 0/393/391 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 10 'sin.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 602 lfor 0/529/527 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 11 'sin.rgw.buckets.data.old' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 603 flags hashpspool stripe_width 0 application rgw pool 12 'sin.rgw.buckets.data' erasure profile data-ec size 6 min_size 5 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 604 flags hashpspool,ec_overwrites stripe_width 16384 application rgw So how I can calculate the pgs? This is my osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 534.38354 root default -5 89.06392 host cephosd-6s01 36 nvme 1.74660 osd.36 up 1.00000 1.00000 0 ssd 14.55289 osd.0 up 1.00000 1.00000 8 ssd 14.55289 osd.8 up 1.00000 1.00000 15 ssd 14.55289 osd.15 up 1.00000 1.00000 18 ssd 14.55289 osd.18 up 1.00000 1.00000 24 ssd 14.55289 osd.24 up 1.00000 1.00000 30 ssd 14.55289 osd.30 up 1.00000 1.00000 -3 89.06392 host cephosd-6s02 37 nvme 1.74660 osd.37 up 1.00000 1.00000 1 ssd 14.55289 osd.1 up 1.00000 1.00000 11 ssd 14.55289 osd.11 up 1.00000 1.00000 17 ssd 14.55289 osd.17 up 1.00000 1.00000 23 ssd 14.55289 osd.23 up 1.00000 1.00000 28 ssd 14.55289 osd.28 up 1.00000 1.00000 35 ssd 14.55289 osd.35 up 1.00000 1.00000 -11 89.06392 host cephosd-6s03 41 nvme 1.74660 osd.41 up 1.00000 1.00000 2 ssd 14.55289 osd.2 up 1.00000 1.00000 6 ssd 14.55289 osd.6 up 1.00000 1.00000 13 ssd 14.55289 osd.13 up 1.00000 1.00000 19 ssd 14.55289 osd.19 up 1.00000 1.00000 26 ssd 14.55289 osd.26 up 1.00000 1.00000 32 ssd 14.55289 osd.32 up 1.00000 1.00000 -13 89.06392 host cephosd-6s04 38 nvme 1.74660 osd.38 up 1.00000 1.00000 5 ssd 14.55289 osd.5 up 1.00000 1.00000 7 ssd 14.55289 osd.7 up 1.00000 1.00000 14 ssd 14.55289 osd.14 up 1.00000 1.00000 20 ssd 14.55289 osd.20 up 1.00000 1.00000 25 ssd 14.55289 osd.25 up 1.00000 1.00000 31 ssd 14.55289 osd.31 up 1.00000 1.00000 -9 89.06392 host cephosd-6s05 40 nvme 1.74660 osd.40 up 1.00000 1.00000 3 ssd 14.55289 osd.3 up 1.00000 1.00000 10 ssd 14.55289 osd.10 up 1.00000 1.00000 12 ssd 14.55289 osd.12 up 1.00000 1.00000 21 ssd 14.55289 osd.21 up 1.00000 1.00000 29 ssd 14.55289 osd.29 up 1.00000 1.00000 33 ssd 14.55289 osd.33 up 1.00000 1.00000 -7 89.06392 host cephosd-6s06 39 nvme 1.74660 osd.39 up 1.00000 1.00000 4 ssd 14.55289 osd.4 up 1.00000 1.00000 9 ssd 14.55289 osd.9 up 1.00000 1.00000 16 ssd 14.55289 osd.16 up 1.00000 1.00000 22 ssd 14.55289 osd.22 up 1.00000 1.00000 27 ssd 14.55289 osd.27 up 1.00000 1.00000 34 ssd 14.55289 osd.34 up 1.00000 1.00000 This is the crush rules: [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "replicated_nvme", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~nvme" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "replicated_ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -2, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 3, "rule_name": "sin.rgw.buckets.data.new", "ruleset": 3, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~ssd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] So everything else rather than the data pool are on SSD and nvme with replica 3. If I calculate the pg in the ec like 36osd*100/6=600 which means the max pg in the EC pool is 512? But how this affect the SSD replica pools then? This is the EC pool definition: crush-device-class=ssd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 5