May 2023 - ceph-users - lists.ceph.io

ceph status is warning with "266 pgs not deep-scrubbed in time"

by Louis Koo

ceph version is: v16.2.10 But I had close the ration with "ceph config set mon mon_warn_pg_not_deep_scrubbed_ratio 0" show the prints: [root@smd-node01 deeproute]# ceph config get mon mon_warn_pg_not_deep_scrubbed_ratio 0.000000

1 year

1
0
0 0

Change in DMARC handling for the list

by Dan Mick

Several users have complained for some time that our DMARC/DKIM handling is not correct. I've recently had time to go study DMARC, DKIM, SPF, SRS, and other tasty morsels of initialisms, and have thus made a change to how Mailman handles DKIM signatures for the list: If a domain advertises that it will reject or quarantine messages that fail DKIM (through its DMARC policy in the DNS text record _dmarc.<domain>), the message will be rewritten to be "From" ceph.io, and SPF should be correct. I do not know if it will regenerate a DKIM signature in that case for what is now it's own message. The From: address will say something like "From Original Sender via ceph-users <ceph-users(a)ceph.io> so it's somewhat clear who first sent the message, and Reply-To will be set to Original Sender. Again, this will only happen for senders from domains that advertise a strict DMARC policy. This does not include gmail.com, surprisingly. Let me know if you notice anything that seems to have gotten worse. Next on the list is to investigate if DKIM-signing outbound messages, or at least ones that don't already have an ARC-Seal, is appropriate and/or workable.

1 year

2
1
0 0

ERROR: Distro uos version 20 not supported

by Ben

Hi, This seems not very relevant since all ceph components are running in containers. Any ideas to get over this issue? Any other ideas or suggestions on this kind of deployment? sudo ./cephadm --image 10.21.22.1:5000/ceph:v17.2.5-20230316 --docker bootstrap --mon-ip 10.21.22.1 --skip-monitoring-stack Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... No time sync service is running; checked for ['chrony.service', 'chronyd.service', 'systemd-timesyncd.service', 'ntpd.service', 'ntp.service', 'ntpsec.service', 'openntpd.service'] ERROR: Distro uos version 20 not supported uname -a Linux aaaa 4.19.0-91.82.42.uelc20.x86_64 #1 SMP Sat May 15 13:50:04 CST 2021 x86_64 x86_64 x86_64 GNU/Linux Thank you in advance Ben

1 year

1
1
0 0

pg deep-scrub issue

by Peter

Dear all, I am writing to seek your assistance in resolving an issue with my Ceph cluster. Currently, the cluster is experiencing a problem where the number of Placement Groups (PGs) that need to undergo deepscrub and regular scrub cannot be completed in a timely manner. I have noticed that these PGs appear to be concentrated on a single OSD. I am seeking your guidance on how to address this issue and would appreciate any insights or suggestions you may have. Please find attached the Ceph health detail output for your reference. HEALTH_WARN 13 pgs not deep-scrubbed in time; 7 pgs not scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 13 pgs not deep-scrubbed in time pg 4.426 not deep-scrubbed since 2023-04-22T10:00:21.529716+0800 pg 4.f0 not deep-scrubbed since 2023-04-22T04:55:17.868881+0800 pg 4.b9 not deep-scrubbed since 2023-04-22T16:47:25.219603+0800 pg 4.87 not deep-scrubbed since 2023-04-22T20:01:02.508600+0800 pg 4.31 not deep-scrubbed since 2023-04-23T00:27:39.299893+0800 pg 4.5b9 not deep-scrubbed since 2023-04-19T21:03:47.041934+0800 pg 4.68a not deep-scrubbed since 2023-04-21T19:52:39.251293+0800 pg 4.6a4 not deep-scrubbed since 2023-04-22T16:20:51.078431+0800 pg 4.6ec not deep-scrubbed since 2023-04-21T11:20:33.661595+0800 pg 4.7a4 not deep-scrubbed since 2023-04-20T22:30:44.506420+0800 pg 4.7a2 not deep-scrubbed since 2023-04-16T12:05:56.586205+0800 pg 4.7b4 not deep-scrubbed since 2023-04-17T15:50:10.595292+0800 pg 4.7c8 not deep-scrubbed since 2023-04-19T15:10:12.673655+0800 [WRN] PG_NOT_SCRUBBED: 7 pgs not scrubbed in time pg 4.31e not scrubbed since 2023-04-24T13:34:26.103257+0800 pg 4.5b9 not scrubbed since 2023-04-24T07:20:53.891175+0800 pg 4.68a not scrubbed since 2023-04-24T03:37:58.070854+0800 pg 4.7a4 not scrubbed since 2023-04-24T02:55:25.912789+0800 pg 4.7b4 not scrubbed since 2023-04-24T10:04:46.889422+0800 pg 4.7c8 not scrubbed since 2023-04-24T13:36:07.284271+0800 pg 4.7d2 not scrubbed since 2023-04-24T14:47:19.365551+0800 Peter

1 year

3
3
0 0

16.2.13 pacific QE validation status

by Yuri Weinstein

Details of this release are summarized here: https://tracker.ceph.com/issues/59542#note-1 Release Notes - TBD Seeking approvals for: smoke - Radek, Laura rados - Radek, Laura rook - Sébastien Han cephadm - Adam K dashboard - Ernesto rgw - Casey rbd - Ilya krbd - Ilya fs - Venky, Patrick upgrade/octopus-x (pacific) - Laura (look the same as in 16.2.8) upgrade/pacific-p2p - Laura powercycle - Brad (SELinux denials) ceph-volume - Guillaume, Adam K Thx YuriW

1 year

10
26
0 0

rgw service fails to start with zone not found

by Adiga, Anantha

Hi, An existing multisite configuration was removed. But the radosgw services still see the old zone name and fail to start. journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs ........... May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 deferred set uid:gid to 167:167 (ceph:ceph) May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 framework: beast May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 framework conf key: port, val: 8080 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 1 radosgw_Main not setting numa affinity May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740 1 D3N datacache enabled: 0 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: could not find zone (fl2site2) May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: failed to start notify service> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: failed to init services (ret=(> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.921+0000 7f2f47634740 -1 Couldn't init storage provider (RADOS) May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs.service: Main pr> May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs.service: Failed Here is the current configuration root@fl31ca104ja0201:/# radosgw-admin period get { "id": "729f7cef-6340-4750-b3ae-9164177c0df3", "epoch": 1, "predecessor_uuid": "1d124ad5-57b0-41de-8def-823bd40f72aa", "sync_status": [], "period_map": { "id": "729f7cef-6340-4750-b3ae-9164177c0df3", "zonegroups": [ { "id": "21b8306c-be43-4567-a0c5-74ab69937535", "name": "default", "api_name": "default", "is_master": "true", "endpoints": [], "hostnames": [], "hostnames_s3website": [], "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3", "zones": [ { "id": "ff468492-8a14-414b-a8cf-7c20ab699af3", "name": "default", "endpoints": [], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312", "sync_policy": { "groups": [] } } ], "short_zone_ids": [ { "key": "ff468492-8a14-414b-a8cf-7c20ab699af3", "val": 3883136521 } ] }, "master_zonegroup": "21b8306c-be43-4567-a0c5-74ab69937535", "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "bucket_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "anonymous_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false } }, "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312", "realm_name": "default", "realm_epoch": 2 } root@fl31ca104ja0201:/# radosgw-admin realm list { "default_info": "8999e04c-17a4-4150-8845-cecd6672d312", "realms": [ "default" ] } root@fl31ca104ja0201:/# radosgw-admin zonegroup list { "default_info": "21b8306c-be43-4567-a0c5-74ab69937535", "zonegroups": [ "default" ] } root@fl31ca104ja0201:/# radosgw-admin zone list { "default_info": "", "zones": [ "default" ] } Where is the radosgw service getting the old zone info from? How to fix this configuration? Thank you, Anantha

1 year

3
4
0 0

Ceph Host offline after performing dnf upgrade on RHEL 8.7 host

by Mevludin Blazevic

Hi all, after I performed a minor RHEL package upgrade (8.7 -> 8.7) in one of our Ceph hosts, I get a Ceph warning describing that cephadm "Can't communicate with remote host `...`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory, although Python3 is installed. However, I figured out that these packages were updated: Installing: kernel 4.18.0-425.19.2.el8_7 kernel-core 4.18.0-425.19.2.el8_7 kernel-modules 4.18.0-425.19.2.el8_7 Upgrading: NetworkManager 1:1.40.0-6.el8_7 NetworkManager-config-server 1:1.40.0-6.el8_7 NetworkManager-initscripts-updown 1:1.40.0-6.el8_7 NetworkManager-libnm 1:1.40.0-6.el8_7 NetworkManager-team 1:1.40.0-6.el8_7 NetworkManager-tui 1:1.40.0-6.el8_7 authselect 1.2.5-2.el8_7 authselect-compat 1.2.5-2.el8_7 authselect-libs 1.2.5-2.el8_7 bpftool 4.18.0-425.19.2.el8_7 buildah 1:1.27.3-1.module+el8.7.0+17824+66a0202b cockpit-podman 53-1.module+el8.7.0+17824+66a0202b conmon 3:2.1.4-1.module+el8.7.0+17824+66a0202b container-selinux 2:2.189.0-1.module+el8.7.0+17824+66a0202b containernetworking-plugins 1:1.1.1-3.module+el8.7.0+17824+66a0202b containers-common 2:1-46.module+el8.7.0+17824+66a0202b criu 3.15-3.module+el8.7.0+17824+66a0202b curl 7.61.1-25.el8_7.3 dbus 1:1.12.8-23.el8_7.1 dbus-common 1:1.12.8-23.el8_7.1 dbus-daemon 1:1.12.8-23.el8_7.1 dbus-libs 1:1.12.8-23.el8_7.1 dbus-tools 1:1.12.8-23.el8_7.1 device-mapper-multipath 0.8.4-28.el8_7.3 device-mapper-multipath-libs 0.8.4-28.el8_7.3 dhcp-client 12:4.3.6-48.el8_7.1 dhcp-common 12:4.3.6-48.el8_7.1 dhcp-libs 12:4.3.6-48.el8_7.1 dracut 049-218.git20221019.el8_7 dracut-config-rescue 049-218.git20221019.el8_7 dracut-network 049-218.git20221019.el8_7 dracut-squash 049-218.git20221019.el8_7 emacs-filesystem 1:26.1-7.el8_7.1 epel-release 8-19.el8 expat 2.2.5-10.el8_7.1 fuse-overlayfs 1.9-1.module+el8.7.0+17824+66a0202b gnutls 3.6.16-6.el8_7 grub2-common 1:2.02-142.el8_7.3 grub2-pc 1:2.02-142.el8_7.3 grub2-pc-modules 1:2.02-142.el8_7.3 grub2-tools 1:2.02-142.el8_7.3 grub2-tools-efi 1:2.02-142.el8_7.3 grub2-tools-extra 1:2.02-142.el8_7.3 grub2-tools-minimal 1:2.02-142.el8_7.3 insights-client 3.1.7-9.el8_7 iptables 1.8.4-23.el8_7.1 iptables-ebtables 1.8.4-23.el8_7.1 iptables-libs 1.8.4-23.el8_7.1 iwl100-firmware 39.31.5.1-111.el8_7.1 iwl1000-firmware 1:39.31.5.1-111.el8_7.1 iwl105-firmware 18.168.6.1-111.el8_7.1 iwl135-firmware 18.168.6.1-111.el8_7.1 iwl2000-firmware 18.168.6.1-111.el8_7.1 iwl2030-firmware 18.168.6.1-111.el8_7.1 iwl3160-firmware 1:25.30.13.0-111.el8_7.1 iwl5000-firmware 8.83.5.1_1-111.el8_7.1 iwl5150-firmware 8.24.2.2-111.el8_7.1 iwl6000-firmware 9.221.4.1-111.el8_7.1 iwl6000g2a-firmware 18.168.6.1-111.el8_7.1 iwl6000g2b-firmware 18.168.6.1-111.el8_7.1 iwl6050-firmware 41.28.5.1-111.el8_7.1 iwl7260-firmware 1:25.30.13.0-111.el8_7.1 kernel-tools 4.18.0-425.19.2.el8_7 kernel-tools-libs 4.18.0-425.19.2.el8_7 kmod-kvdo 6.2.7.17-88.el8_7 kpartx 0.8.4-28.el8_7.3 libblkid 2.32.1-39.el8_7 libcurl 7.61.1-25.el8_7.3 libertas-usb8388-firmware 2:20220726-111.git150864a4.el8_7 libfdisk 2.32.1-39.el8_7 libgcc 8.5.0-16.el8_7 libgomp 8.5.0-16.el8_7 libipa_hbac 2.7.3-4.el8_7.3 libksba 1.3.5-9.el8_7 libmount 2.32.1-39.el8_7 libnfsidmap 1:2.3.3-57.el8_7.1 libslirp 4.4.0-1.module+el8.7.0+17824+66a0202b libsmartcols 2.32.1-39.el8_7 libsmbclient 4.16.4-6.el8_7 libsolv 0.7.20-4.el8_7 libsss_autofs 2.7.3-4.el8_7.3 libsss_certmap 2.7.3-4.el8_7.3 libsss_idmap 2.7.3-4.el8_7.3 libsss_nss_idmap 2.7.3-4.el8_7.3 libsss_sudo 2.7.3-4.el8_7.3 libstdc++ 8.5.0-16.el8_7 libtasn1 4.13-4.el8_7 libuuid 2.32.1-39.el8_7 libwbclient 4.16.4-6.el8_7 libxml2 2.9.7-15.el8_7.1 linux-firmware 20220726-111.git150864a4.el8_7 nss 3.79.0-11.el8_7 nss-softokn 3.79.0-11.el8_7 nss-softokn-freebl 3.79.0-11.el8_7 nss-sysinit 3.79.0-11.el8_7 nss-util 3.79.0-11.el8_7 openssh 8.0p1-17.el8_7 openssh-clients 8.0p1-17.el8_7 openssh-server 8.0p1-17.el8_7 openssl 1:1.1.1k-9.el8_7 openssl-libs 1:1.1.1k-9.el8_7 platform-python 3.6.8-48.el8_7.1 platform-python-setuptools 39.2.0-6.el8_7.1 podman 3:4.2.0-8.module+el8.7.0+17824+66a0202b podman-catatonit 3:4.2.0-8.module+el8.7.0+17824+66a0202b python3-libs 3.6.8-48.el8_7.1 python3-libxml2 2.9.7-15.el8_7.1 python3-perf 4.18.0-425.19.2.el8_7 python3-setuptools 39.2.0-6.el8_7.1 python3-setuptools-wheel 39.2.0-6.el8_7.1 python3-sssdconfig 2.7.3-4.el8_7.3 rhc 1:0.2.1-12.el8_7 rsync 3.1.3-19.el8_7.1 runc 1:1.1.4-1.module+el8.7.0+17824+66a0202b samba-client-libs 4.16.4-6.el8_7 samba-common 4.16.4-6.el8_7 samba-common-libs 4.16.4-6.el8_7 selinux-policy 3.14.3-108.el8_7.2 selinux-policy-targeted 3.14.3-108.el8_7.2 slirp4netns 1.2.0-2.module+el8.7.0+17824+66a0202b sos 4.5.1-3.el8 sqlite 3.26.0-17.el8_7 sqlite-libs 3.26.0-17.el8_7 sssd 2.7.3-4.el8_7.3 sssd-ad 2.7.3-4.el8_7.3 sssd-client 2.7.3-4.el8_7.3 sssd-common 2.7.3-4.el8_7.3 sssd-common-pac 2.7.3-4.el8_7.3 sssd-ipa 2.7.3-4.el8_7.3 sssd-kcm 2.7.3-4.el8_7.3 sssd-krb5 2.7.3-4.el8_7.3 sssd-krb5-common 2.7.3-4.el8_7.3 sssd-ldap 2.7.3-4.el8_7.3 sssd-nfs-idmap 2.7.3-4.el8_7.3 sssd-proxy 2.7.3-4.el8_7.3 sudo 1.8.29-8.el8_7.1 systemd 239-68.el8_7.4 systemd-libs 239-68.el8_7.4 systemd-pam 239-68.el8_7.4 systemd-udev 239-68.el8_7.4 tar 2:1.30-6.el8_7.1 tzdata 2023c-1.el8 util-linux 2.32.1-39.el8_7 util-linux-user 2.32.1-39.el8_7 zlib 1.2.11-21.el8_7 Is something odd with the updated Python3 packages? Best, Mevludin

1 year

1
1
0 0

Os changed to Ubuntu, device class not shown

by Szabo, Istvan (Agoda)

Hi, We have an octopus cluster where we want to move from centos to Ubuntu, after activate all the osd, class is not shown in ceph osd tree. However ceph-volume list shows the crush device class :/ Should I just add it or? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 year

1
0
0 0

Testing to simulate real-world usage scenarios

by Michal Strnad

Hi everyone, how do you test features that are used by your users before updating/upgrading to a newer version of Ceph? For example, if you have users or customers using RBD images with special sets of permissions/caps, or users/customers using S3 with versioning... None of these should be broken by the new code. Do you use any framework, set of scripts, or anything else? So far, I have only found Teuthology for integration tests and then for S3 [https://github.com/ceph/s3-tests]. Thank you Michal

1 year

1
0
0 0

splitting client and replication traffic

by Justin Ryan

hiya folks! i am excited to be learning about ceph in my homelab, where i have a couple trios of machines service, currently, a cephfs, but i’m looking fw to using ceph for block devices for vms, etc.. my phyiscal machines have multiple network interfaces, one trio of them are ARM machines with up to 4x SFP+ 10GBe interfaces. I have a couple of 10GBe switches - one which has exactly three ports I can connect these nodes to for general distribution to the rest of the network, and another… I would like to configure my Ceph cluster to replicate over a different network interface than it serves clients over.. Is this a common configuration, and/or can anyone provide me some guidance?! Thanks in advance! Best! J -- Justin Alan Ryan

1 year

2
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users May 2023