- ceph-users - lists.ceph.io

rgw service fails to start with zone not found
by Adiga, Anantha 08 May '23

08 May '23

Hi, An existing multisite configuration was removed. But the radosgw services still see the old zone name and fail to start. journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs ........... May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 deferred set uid:gid to 167:167 (ceph:ceph) May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 framework: beast May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 framework conf key: port, val: 8080 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 1 radosgw_Main not setting numa affinity May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740 1 D3N datacache enabled: 0 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: could not find zone (fl2site2) May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: failed to start notify service> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: failed to init services (ret=(> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.921+0000 7f2f47634740 -1 Couldn't init storage provider (RADOS) May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs.service: Main pr> May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs.service: Failed Here is the current configuration root@fl31ca104ja0201:/# radosgw-admin period get { "id": "729f7cef-6340-4750-b3ae-9164177c0df3", "epoch": 1, "predecessor_uuid": "1d124ad5-57b0-41de-8def-823bd40f72aa", "sync_status": [], "period_map": { "id": "729f7cef-6340-4750-b3ae-9164177c0df3", "zonegroups": [ { "id": "21b8306c-be43-4567-a0c5-74ab69937535", "name": "default", "api_name": "default", "is_master": "true", "endpoints": [], "hostnames": [], "hostnames_s3website": [], "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3", "zones": [ { "id": "ff468492-8a14-414b-a8cf-7c20ab699af3", "name": "default", "endpoints": [], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312", "sync_policy": { "groups": [] } } ], "short_zone_ids": [ { "key": "ff468492-8a14-414b-a8cf-7c20ab699af3", "val": 3883136521 } ] }, "master_zonegroup": "21b8306c-be43-4567-a0c5-74ab69937535", "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "bucket_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "anonymous_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false } }, "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312", "realm_name": "default", "realm_epoch": 2 } root@fl31ca104ja0201:/# radosgw-admin realm list { "default_info": "8999e04c-17a4-4150-8845-cecd6672d312", "realms": [ "default" ] } root@fl31ca104ja0201:/# radosgw-admin zonegroup list { "default_info": "21b8306c-be43-4567-a0c5-74ab69937535", "zonegroups": [ "default" ] } root@fl31ca104ja0201:/# radosgw-admin zone list { "default_info": "", "zones": [ "default" ] } Where is the radosgw service getting the old zone info from? How to fix this configuration? Thank you, Anantha

3 4

Ceph Host offline after performing dnf upgrade on RHEL 8.7 host
by Mevludin Blazevic 08 May '23

08 May '23

Hi all, after I performed a minor RHEL package upgrade (8.7 -> 8.7) in one of our Ceph hosts, I get a Ceph warning describing that cephadm "Can't communicate with remote host `...`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory, although Python3 is installed. However, I figured out that these packages were updated: Installing: kernel 4.18.0-425.19.2.el8_7 kernel-core 4.18.0-425.19.2.el8_7 kernel-modules 4.18.0-425.19.2.el8_7 Upgrading: NetworkManager 1:1.40.0-6.el8_7 NetworkManager-config-server 1:1.40.0-6.el8_7 NetworkManager-initscripts-updown 1:1.40.0-6.el8_7 NetworkManager-libnm 1:1.40.0-6.el8_7 NetworkManager-team 1:1.40.0-6.el8_7 NetworkManager-tui 1:1.40.0-6.el8_7 authselect 1.2.5-2.el8_7 authselect-compat 1.2.5-2.el8_7 authselect-libs 1.2.5-2.el8_7 bpftool 4.18.0-425.19.2.el8_7 buildah 1:1.27.3-1.module+el8.7.0+17824+66a0202b cockpit-podman 53-1.module+el8.7.0+17824+66a0202b conmon 3:2.1.4-1.module+el8.7.0+17824+66a0202b container-selinux 2:2.189.0-1.module+el8.7.0+17824+66a0202b containernetworking-plugins 1:1.1.1-3.module+el8.7.0+17824+66a0202b containers-common 2:1-46.module+el8.7.0+17824+66a0202b criu 3.15-3.module+el8.7.0+17824+66a0202b curl 7.61.1-25.el8_7.3 dbus 1:1.12.8-23.el8_7.1 dbus-common 1:1.12.8-23.el8_7.1 dbus-daemon 1:1.12.8-23.el8_7.1 dbus-libs 1:1.12.8-23.el8_7.1 dbus-tools 1:1.12.8-23.el8_7.1 device-mapper-multipath 0.8.4-28.el8_7.3 device-mapper-multipath-libs 0.8.4-28.el8_7.3 dhcp-client 12:4.3.6-48.el8_7.1 dhcp-common 12:4.3.6-48.el8_7.1 dhcp-libs 12:4.3.6-48.el8_7.1 dracut 049-218.git20221019.el8_7 dracut-config-rescue 049-218.git20221019.el8_7 dracut-network 049-218.git20221019.el8_7 dracut-squash 049-218.git20221019.el8_7 emacs-filesystem 1:26.1-7.el8_7.1 epel-release 8-19.el8 expat 2.2.5-10.el8_7.1 fuse-overlayfs 1.9-1.module+el8.7.0+17824+66a0202b gnutls 3.6.16-6.el8_7 grub2-common 1:2.02-142.el8_7.3 grub2-pc 1:2.02-142.el8_7.3 grub2-pc-modules 1:2.02-142.el8_7.3 grub2-tools 1:2.02-142.el8_7.3 grub2-tools-efi 1:2.02-142.el8_7.3 grub2-tools-extra 1:2.02-142.el8_7.3 grub2-tools-minimal 1:2.02-142.el8_7.3 insights-client 3.1.7-9.el8_7 iptables 1.8.4-23.el8_7.1 iptables-ebtables 1.8.4-23.el8_7.1 iptables-libs 1.8.4-23.el8_7.1 iwl100-firmware 39.31.5.1-111.el8_7.1 iwl1000-firmware 1:39.31.5.1-111.el8_7.1 iwl105-firmware 18.168.6.1-111.el8_7.1 iwl135-firmware 18.168.6.1-111.el8_7.1 iwl2000-firmware 18.168.6.1-111.el8_7.1 iwl2030-firmware 18.168.6.1-111.el8_7.1 iwl3160-firmware 1:25.30.13.0-111.el8_7.1 iwl5000-firmware 8.83.5.1_1-111.el8_7.1 iwl5150-firmware 8.24.2.2-111.el8_7.1 iwl6000-firmware 9.221.4.1-111.el8_7.1 iwl6000g2a-firmware 18.168.6.1-111.el8_7.1 iwl6000g2b-firmware 18.168.6.1-111.el8_7.1 iwl6050-firmware 41.28.5.1-111.el8_7.1 iwl7260-firmware 1:25.30.13.0-111.el8_7.1 kernel-tools 4.18.0-425.19.2.el8_7 kernel-tools-libs 4.18.0-425.19.2.el8_7 kmod-kvdo 6.2.7.17-88.el8_7 kpartx 0.8.4-28.el8_7.3 libblkid 2.32.1-39.el8_7 libcurl 7.61.1-25.el8_7.3 libertas-usb8388-firmware 2:20220726-111.git150864a4.el8_7 libfdisk 2.32.1-39.el8_7 libgcc 8.5.0-16.el8_7 libgomp 8.5.0-16.el8_7 libipa_hbac 2.7.3-4.el8_7.3 libksba 1.3.5-9.el8_7 libmount 2.32.1-39.el8_7 libnfsidmap 1:2.3.3-57.el8_7.1 libslirp 4.4.0-1.module+el8.7.0+17824+66a0202b libsmartcols 2.32.1-39.el8_7 libsmbclient 4.16.4-6.el8_7 libsolv 0.7.20-4.el8_7 libsss_autofs 2.7.3-4.el8_7.3 libsss_certmap 2.7.3-4.el8_7.3 libsss_idmap 2.7.3-4.el8_7.3 libsss_nss_idmap 2.7.3-4.el8_7.3 libsss_sudo 2.7.3-4.el8_7.3 libstdc++ 8.5.0-16.el8_7 libtasn1 4.13-4.el8_7 libuuid 2.32.1-39.el8_7 libwbclient 4.16.4-6.el8_7 libxml2 2.9.7-15.el8_7.1 linux-firmware 20220726-111.git150864a4.el8_7 nss 3.79.0-11.el8_7 nss-softokn 3.79.0-11.el8_7 nss-softokn-freebl 3.79.0-11.el8_7 nss-sysinit 3.79.0-11.el8_7 nss-util 3.79.0-11.el8_7 openssh 8.0p1-17.el8_7 openssh-clients 8.0p1-17.el8_7 openssh-server 8.0p1-17.el8_7 openssl 1:1.1.1k-9.el8_7 openssl-libs 1:1.1.1k-9.el8_7 platform-python 3.6.8-48.el8_7.1 platform-python-setuptools 39.2.0-6.el8_7.1 podman 3:4.2.0-8.module+el8.7.0+17824+66a0202b podman-catatonit 3:4.2.0-8.module+el8.7.0+17824+66a0202b python3-libs 3.6.8-48.el8_7.1 python3-libxml2 2.9.7-15.el8_7.1 python3-perf 4.18.0-425.19.2.el8_7 python3-setuptools 39.2.0-6.el8_7.1 python3-setuptools-wheel 39.2.0-6.el8_7.1 python3-sssdconfig 2.7.3-4.el8_7.3 rhc 1:0.2.1-12.el8_7 rsync 3.1.3-19.el8_7.1 runc 1:1.1.4-1.module+el8.7.0+17824+66a0202b samba-client-libs 4.16.4-6.el8_7 samba-common 4.16.4-6.el8_7 samba-common-libs 4.16.4-6.el8_7 selinux-policy 3.14.3-108.el8_7.2 selinux-policy-targeted 3.14.3-108.el8_7.2 slirp4netns 1.2.0-2.module+el8.7.0+17824+66a0202b sos 4.5.1-3.el8 sqlite 3.26.0-17.el8_7 sqlite-libs 3.26.0-17.el8_7 sssd 2.7.3-4.el8_7.3 sssd-ad 2.7.3-4.el8_7.3 sssd-client 2.7.3-4.el8_7.3 sssd-common 2.7.3-4.el8_7.3 sssd-common-pac 2.7.3-4.el8_7.3 sssd-ipa 2.7.3-4.el8_7.3 sssd-kcm 2.7.3-4.el8_7.3 sssd-krb5 2.7.3-4.el8_7.3 sssd-krb5-common 2.7.3-4.el8_7.3 sssd-ldap 2.7.3-4.el8_7.3 sssd-nfs-idmap 2.7.3-4.el8_7.3 sssd-proxy 2.7.3-4.el8_7.3 sudo 1.8.29-8.el8_7.1 systemd 239-68.el8_7.4 systemd-libs 239-68.el8_7.4 systemd-pam 239-68.el8_7.4 systemd-udev 239-68.el8_7.4 tar 2:1.30-6.el8_7.1 tzdata 2023c-1.el8 util-linux 2.32.1-39.el8_7 util-linux-user 2.32.1-39.el8_7 zlib 1.2.11-21.el8_7 Is something odd with the updated Python3 packages? Best, Mevludin

1 1

Os changed to Ubuntu, device class not shown
by Szabo, Istvan (Agoda) 08 May '23

08 May '23

Hi, We have an octopus cluster where we want to move from centos to Ubuntu, after activate all the osd, class is not shown in ceph osd tree. However ceph-volume list shows the crush device class :/ Should I just add it or? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Testing to simulate real-world usage scenarios
by Michal Strnad 07 May '23

07 May '23

Hi everyone, how do you test features that are used by your users before updating/upgrading to a newer version of Ceph? For example, if you have users or customers using RBD images with special sets of permissions/caps, or users/customers using S3 with versioning... None of these should be broken by the new code. Do you use any framework, set of scripts, or anything else? So far, I have only found Teuthology for integration tests and then for S3 [https://github.com/ceph/s3-tests] Thank you Michal

1 0

splitting client and replication traffic
by Justin Ryan 06 May '23

06 May '23

hiya folks! i am excited to be learning about ceph in my homelab, where i have a couple trios of machines service, currently, a cephfs, but i’m looking fw to using ceph for block devices for vms, etc.. my phyiscal machines have multiple network interfaces, one trio of them are ARM machines with up to 4x SFP+ 10GBe interfaces. I have a couple of 10GBe switches - one which has exactly three ports I can connect these nodes to for general distribution to the rest of the network, and another… I would like to configure my Ceph cluster to replicate over a different network interface than it serves clients over.. Is this a common configuration, and/or can anyone provide me some guidance?! Thanks in advance! Best! J -- Justin Alan Ryan

2 1

Unable to restart mds - mds crashes almost immediately after finishing recovery
by Emmanuel Jaep 05 May '23

05 May '23

Hi, I just inherited a ceph storage. Therefore, my level of confidence with the tool is certainly less than ideal. We currently have an mds server that refuses to come back online. While reviewing the logs, I can see that, upon mds start, the recovery goes well: ``` -10> 2023-05-03T08:12:43.632+0200 7f345d00b700 1 mds.4.2638711 cluster recovered. 12: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] ``` However, rights after this message, ceph handles a couple of clients request: ``` -9> 2023-05-03T08:12:43.632+0200 7f345d00b700 4 mds.4.2638711 set_osd_epoch_barrier: epoch=249241 -8> 2023-05-03T08:12:43.632+0200 7f3459003700 2 mds.4.cache Memory usage: total 2739784, rss 2321188, heap 348412, baseline 315644, 0 / 765023 inodes have caps, 0 caps, 0 caps per inode -7> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.108396030:57271 lookup #0x70001516236/012385530.npy 2023-05-02T20:37:19.675666+0200 RETRY=6 caller_uid=135551, caller_gid=11157{0,4,27,11157,}) v5 -6> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.104073212:5109945 readdir #0x70001516236 2023-05-02T20:36:29.517066+0200 RETRY=6 caller_uid=180090, caller_gid=11157{0,4,27,11157,}) v5 -5> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.104288735:3008344 readdir #0x70001516236 2023-05-02T20:36:29.520801+0200 RETRY=6 caller_uid=135551, caller_gid=11157{0,4,27,11157,}) v5 -4> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.8558540:46306346 readdir #0x700019ba15e 2023-05-01T21:26:34.303697+0200 RETRY=49 caller_uid=0, caller_gid=0{}) v2 -3> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.96913903:2156912 create #0x1000b37db9a/street-photo-3.png 2023-05-01T17:27:37.454042+0200 RETRY=59 caller_uid=271932, caller_gid=30034{}) v2 -2> 2023-05-03T08:12:43.688+0200 7f345d00b700 5 mds.icadmin006 handle_mds_map old map epoch 2638715 <= 2638715, discarding ``` and crashes: ``` -1> 2023-05-03T08:12:43.692+0200 7f345d00b700 -1 /build/ceph-16.2.10/src/mds/Server.cc: In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f345d00b700 time 2023-05-03T08:12:43.694660+0200 /build/ceph-16.2.10/src/mds/Server.cc: 4240: FAILED ceph_assert(cur->is_auth()) ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f3462533d65] 2: /usr/lib/ceph/libceph-common.so.2(+0x265f6d) [0x7f3462533f6d] 3: (Server::handle_client_open(boost::intrusive_ptr<MDRequestImpl>&)+0x1834) [0x558323c89f04] 4: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0x28f) [0x558323c925ef] 5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xa45) [0x558323cc3575] 6: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x3d) [0x558323d7460d] 7: (MDSContext::complete(int)+0x61) [0x558323f68681] 8: (MDCache::_open_remote_dentry_finish(CDentry*, inodeno_t, MDSContext*, bool, int)+0x3e) [0x558323d3edce] 9: (C_MDC_OpenRemoteDentry::finish(int)+0x3e) [0x558323de6cce] 10: (MDSContext::complete(int)+0x61) [0x558323f68681] 11: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0xcf) [0x558323d5ff2f] 12: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] 13: (MDSContext::complete(int)+0x61) [0x558323f68681] 14: (MDSRank::_advance_queues()+0x88) [0x558323c23c38] 15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x1fa) [0x558323c24a1a] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5e) [0x558323c254fe] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x558323bfd906] 18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f34627854e0] 19: (DispatchQueue::entry()+0x58f) [0x7f3462782d7f] 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f346284eee1] 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f3462278609] 22: clone() 0> 2023-05-03T08:12:43.700+0200 7f345d00b700 -1 *** Caught signal (Aborted) ** in thread 7f345d00b700 thread_name:ms_dispatch ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0) [0x7f34622843c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x7f3462533dc0] 5: /usr/lib/ceph/libceph-common.so.2(+0x265f6d) [0x7f3462533f6d] 6: (Server::handle_client_open(boost::intrusive_ptr<MDRequestImpl>&)+0x1834) [0x558323c89f04] 7: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0x28f) [0x558323c925ef] 8: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xa45) [0x558323cc3575] 9: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x3d) [0x558323d7460d] 10: (MDSContext::complete(int)+0x61) [0x558323f68681] 11: (MDCache::_open_remote_dentry_finish(CDentry*, inodeno_t, MDSContext*, bool, int)+0x3e) [0x558323d3edce] 12: (C_MDC_OpenRemoteDentry::finish(int)+0x3e) [0x558323de6cce] 13: (MDSContext::complete(int)+0x61) [0x558323f68681] 14: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0xcf) [0x558323d5ff2f] 15: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] 16: (MDSContext::complete(int)+0x61) [0x558323f68681] 17: (MDSRank::_advance_queues()+0x88) [0x558323c23c38] 18: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x1fa) [0x558323c24a1a] 19: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5e) [0x558323c254fe] 20: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x558323bfd906] 21: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f34627854e0] 22: (DispatchQueue::entry()+0x58f) [0x7f3462782d7f] 23: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f346284eee1] 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f3462278609] 25: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 139862749464320 / 139862757857024 / md_submit 139862766249728 / 139862774642432 / MR_Finisher 139862791427840 / PQ_Finisher 139862799820544 / mds_rank_progr 139862808213248 / ms_dispatch 139862841784064 / ceph-mds 139862858569472 / safe_timer 139862875354880 / ms_dispatch 139862892140288 / io_context_pool 139862908925696 / admin_socket 139862917318400 / msgr-worker-2 139862925711104 / msgr-worker-1 139862934103808 / msgr-worker-0 139862951257984 / ceph-mds max_recent 10000 max_new 10000 log_file /var/log/ceph/floki-mds.icadmin006.log --- end dump of recent events --- ``` How could I troubleshoot that further? Thanks in advance for your help, Emmanuel

3 3

osd pause
by Thomas Bennett 05 May '23

05 May '23

Hi, FYI - This might be pedantic, but there does not seem to be any difference between using these two sets of commands: - ceph osd pause / ceph osd unpause - ceph osd set pause / ceph osd unset pause I can see that they both set/unset the pauserd,pausewr flags, but since they don't report anything else, I assume they do exactly the same thing. I also assumed it only stopped reads/writes to the OSDs, but I found this openattic post <https://openattic.org/posts/how-to-do-a-ceph-cluster-maintenanceshutdown/> which had this comment: Pausing the cluster means that you can't see when OSDs come back up again and no map update will happen. I didn't know that but it seems pretty useful knowing. Pausing is mentioned when posts talking about shutting down Ceph cluster for maintenance but often it's added as optional. Does anyone know what is the original intended purpose of pausing and when/why would you use it? Also - can I assume that pause will complete any current write/read ops on OSDs before pausing? Thanks, Tom

1 0

rbd map: corrupt full osdmap (-22) when
by Kamil Madac 05 May '23

05 May '23

Hi, We deployed pacific cluster 16.2.12 with cephadm. We experience following error during rbd map: [Wed May 3 08:59:11 2023] libceph: mon2 (1)[2a00:da8:ffef:1433::]:6789 session established [Wed May 3 08:59:11 2023] libceph: another match of type 1 in addrvec [Wed May 3 08:59:11 2023] libceph: corrupt full osdmap (-22) epoch 200 off 1042 (000000009876284d of 000000000cb24b58-0000000080b70596) [Wed May 3 08:59:11 2023] osdmap: 00000000: 08 07 7d 10 00 00 09 01 5d 09 00 00 a2 22 3b 86 ..}.....]....";. [Wed May 3 08:59:11 2023] osdmap: 00000010: e4 f5 11 ed 99 ee 47 75 ca 3c ad 23 c8 00 00 00 ......Gu.<.#.... [Wed May 3 08:59:11 2023] osdmap: 00000020: 21 68 4a 64 98 d2 5d 2e 84 fd 50 64 d9 3a 48 26 !hJd..]...Pd.:H& [Wed May 3 08:59:11 2023] osdmap: 00000030: 02 00 00 00 01 00 00 00 00 00 00 00 1d 05 71 01 ..............q. .... Linux Kernel is 6.1.13 and the important thing is that we are using ipv6 addresses for connection to ceph nodes. We were able to map rbd from client with kernel 5.10, but in prod environment we are not allowed to use that kernel. What could be the reason for such behavior on newer kernels and how to troubleshoot it? Here is output of ceph osd dump: # ceph osd dump epoch 200 fsid a2223b86-e4f5-11ed-99ee-4775ca3cad23 created 2023-04-27T12:18:41.777900+0000 modified 2023-05-02T12:09:40.642267+0000 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 34 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client jewel require_osd_release pacific stretch_mode_enabled false pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 183 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 'idp' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 48 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd max_osd 3 osd.0 up in weight 1 up_from 176 up_thru 182 down_at 172 last_clean_interval [170,171) [v2:[2a00:da8:ffef:1431::]:6800/805023868,v1:[2a00:da8:ffef:1431::]:6801/805023868,v2: 0.0.0.0:6802/805023868,v1:0.0.0.0:6803/805023868] [v2:[2a00:da8:ffef:1431::]:6804/805023868,v1:[2a00:da8:ffef:1431::]:6805/805023868,v2: 0.0.0.0:6806/805023868,v1:0.0.0.0:6807/805023868] exists,up e8fd0ee2-ea63-4d02-8f36-219d36869078 osd.1 up in weight 1 up_from 136 up_thru 182 down_at 0 last_clean_interval [0,0) [v2:[2a00:da8:ffef:1432::]:6800/2172723816,v1:[2a00:da8:ffef:1432::]:6801/2172723816,v2: 0.0.0.0:6802/2172723816,v1:0.0.0.0:6803/2172723816] [v2:[2a00:da8:ffef:1432::]:6804/2172723816,v1:[2a00:da8:ffef:1432::]:6805/2172723816,v2: 0.0.0.0:6806/2172723816,v1:0.0.0.0:6807/2172723816] exists,up 0b7b5628-9273-4757-85fb-9c16e8441895 osd.2 up in weight 1 up_from 182 up_thru 182 down_at 178 last_clean_interval [123,177) [v2:[2a00:da8:ffef:1433::]:6800/887631330,v1:[2a00:da8:ffef:1433::]:6801/887631330,v2: 0.0.0.0:6802/887631330,v1:0.0.0.0:6803/887631330] [v2:[2a00:da8:ffef:1433::]:6804/887631330,v1:[2a00:da8:ffef:1433::]:6805/887631330,v2: 0.0.0.0:6806/887631330,v1:0.0.0.0:6807/887631330] exists,up 21f8d0d5-6a3f-4f78-96c8-8ec4e4f78a01 Thank you. -- Kamil Madac

3 5

Re: 17.2.6 fs 'ls' ok, but 'cat' 'operation not permitted' puzzle
by Harry G Coin 05 May '23

05 May '23

This problem of inaccessible file systems post upgrade by other than client.admin date back from v14 carries on through v17. It also applies to any case of specifying other than the default pool names for new file systems. Solved because Curt remembered link on this list. (Thanks Curt!) Here's what the official ceph docs ought have provided, for others who hit this. YMMV: IF you have ceph file systems which have data and meta data pools that were specified in the 'ceph fs new' command (meaning not left to the defaults which create them for you), OR you have an existing ceph file system and are upgrading to a new major version of ceph THEN for the documented 'ceph fs authorize...' commands to do as documented (and avoid strange 'operation not permitted' errors when doing file I/O or similar security related problems for all but such as the client.admin user), you must first run: ceph osd pool application set <your metadata pool name> cephfs metadata <your ceph fs filesystem name> and ceph osd pool application set <your data pool name> cephfs data <your ceph fs filesystem name> Otherwise when the OSD's get a request to read or write data (not the directory info, but file data) they won't know which ceph file system name to look up, nevermind the names you may have chosen for the pools, as the 'defaults' themselves changed in the major releases, from data pool=fsname metadata pool=fsname_metadata to data pool=fsname.data and metadata pool=fsname.meta as the ceph revisions came and went. Any setup that just used 'client.admin' for all mounts didn't see the problem as the admin key gave blanket permission. A temporary 'fix' is to change mount requests to the 'client.admin' and associated key. A less drastic but still half-fix is to change the osd cap for your user to just 'caps osd = "allow rw" and delete "tag cephfs data=...." The only documentation I could find for this upgrade security-related ceph-ending catastrophe was in the NFS, not cephfs docs: https://docs.ceph.com/en/latest/cephfs/nfs/ and the Genius level much appreciated pointer from Curt here: On 5/2/23 14:21, Curt wrote: > This thread might be of use, it's an older version of ceph 14, but > might still apply, > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… > ? > > On Tue, May 2, 2023 at 11:06 PM Harry G Coin <hgcoin(a)gmail.com> wrote: > > In 17.2.6 is there a security requirement that pool names > supporting a > ceph fs filesystem match the filesystem name.data for the data and > name.meta for the associated metadata pool? (multiple file systems > are > enabled) > > I have filesystems from older versions with the data pool name > matching > the filesystem and appending _metadata for that, > > and even older filesystems with the pool name as in 'library' and > 'library_metadata' supporting a filesystem called 'libraryfs' > > The pools all have the cephfs tag. > > But using the documented: > > ceph fs authorize libraryfs client.basicuser / rw > > command allows the root user to mount and browse the library > directory > tree, but fails with 'operation not permitted' when even reading > any file. > > However, changing the client.basicuser osd auth to 'allow rw' > instead of > 'allow rw tag...' allows normal operations. > > So: > > [client.basicuser] > key = <key stuff>== > caps mds = "allow rw fsname=libraryfs" > caps mon = "allow r fsname=libraryfs" > caps osd = "allow rw" > > works, but the same with > > caps osd = "allow rw tag cephfs data=libraryfs" > > leads to the 'operation not permitted' on read, or write or any > actual > access. > > It remains a puzzle. Help appreciated! > > Were there upgrade instructions about that, any help pointing me > to them? > > Thanks > > Harry Coin > Rock Stable Systems > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 2

CephFS Scrub Questions
by Chris Palmer 04 May '23

04 May '23

Hi Grateful if someone could clarify some things about CephFS Scrubs: 1) Am I right that a command such as "ceph tell mds.cephfs:0 scrub start / recursive" only triggers a forward scrub (not a backward scrub)? 2) I couldn't find any reference to forward scrubs being done automatically and was wondering whether I should do them using cron? But then I saw an undated (but I think a little elderly) presentation by Greg Farnum that states that "forward scrub...runs continuously in the background". Is that still correct (for Quincy), and if so what controls the frequency? 3) Are backward scrubs always manual, using the 3 cephfs-data-scan phases? 4) Are regular backward scrubs recommended, or only if there is indication of a problem? (With due regard to the amount of time they may take...) Thanks for any advice. Regards, Chris

2 1