- ceph-users - lists.ceph.io

iDRAC 9 version 6.10 shows 0% for write endurance on non-dell drives, work around?
by Drew Weaver 14 Feb '23

14 Feb '23

Hello, After upgrading a lot of iDRAC9 modules to version 6.10 in servers that are involved in a Ceph cluster we noticed that the iDRAC9 shows the write endurance as 0% on any non-certified disk. OMSA still shows the correct remaining write endurance but I am assuming that they are working feverishly to eliminate that too. I opened a support ticket with Dell once this was brought to my attention and they basically told me that I was lucky that it ever worked at all, which I thought was an odd response given that the iDRAC enterprise licenses cost several hundred dollars each. I know that the old Intel Datacenter Tool used to be able to reach through a MegaRAID controller and read the remaining write endurance but that tool is essentially defunct now. What are you folks using to monitor your write endurance on your SSDs that you couldn't buy from Dell because they had a 16 week lead time while the MFG could deliver the drives in 3 days? Thanks, -Drew

2 2

Cephalocon 2023 Amsterdam Call For Proposals Extended to February 19!
by Mike Perez 14 Feb '23

14 Feb '23

CALL FOR PROPOSALS EXTENDED‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ [image: Logo] CALL FOR PROPOSALS EXTENDED <https://events.linuxfoundation.org/cephalocon/program/cfp/> [image: Ceph23-LaunchAssets-v1_snackable-1200x627-A.png] <https://events.linuxfoundation.org/cephalocon/> We heard your feedback on the short notice for the Call for Proposals <https://events.linuxfoundation.org/cephalocon/program/cfp/> and have extended the deadline to Sunday, February 19, at 11:59 pm PST! Cephalocon <https://events.linuxfoundation.org/cephalocon/> is the premier yearly event that brings together the global community of operators, developers, and researchers for Ceph, the open source distributed storage system designed to provide excellent performance, reliability, and scalability. Join new and existing community members worldwide to learn more about Ceph and the project’s future from the developers writing the code and the operators deploying it at scale. SUBMIT YOUR PROPOSAL <https://events.linuxfoundation.org/cephalocon/program/cfp/> Important Dates - *CFP Closes: *Sunday, February 19 at 11:59 pm PST - *CFP Notifications:* Friday, February 24 - *Schedule Announcement:* Wednesday, March 1 - *Presentation Slide Due Date:* Wednesday, April 12 - *Event Dates:* Monday, April 17 – Tuesday, April 18 (Developer Summit Sunday, April 16) Sponsorship Sponsoring Cephalocon <https://events.linuxfoundation.org/cephalocon/sponsor/> is a unique opportunity to gain valuable mindshare with an elite audience of engineers, researchers, and end-users. Building on the success of Ceph Days and virtual Developer Summits, Cephalocon brings together more than 400 attendees from across the globe to showcase Ceph’s history and its future, real-world applications, and of course, highlight vendor solutions. Cephalocon 2023 promises to make for incredible community building, cross-company collaboration, and cutting-edge training. Contact us at sponsorships(a)ceph.foundation to secure your sponsorship, request additional details or discuss custom options. See Sponsorship Prospectus <https://events.linuxfoundation.org/cephalocon/sponsor/> [image: Link icon] <https://ceph.io> [image: Twitter icon] <https://twitter.com/ceph> [image: YouTube icon] <https://youtube.com/c/cephstorage> [image: LinkedIn icon] <https://www.linkedin.com/company/ceph/> [image: Facebook icon] <https://www.facebook.com/cephstorage/> f8xj3pa8x9xf

2 1

PSA: Potential problems in a recent kernel?
by Matthew Booth 14 Feb '23

14 Feb '23

My ceph cluster became unstable yesterday after zincati (CoreOS's auto-updater) updated one of my nodes from 37.20221225.3.0 to 37.20230110.3.1(*). The symptom was slow ops in my cephfs mds which started immediately the OSDs on this node became in and up. Excluding the OSDs on this node worked round the problem. Note that the node is also running a mon and client workloads which use ceph. Also note that the OSD came up and (IIUC) were participating in recovering their data to other OSDs. The problem only started when I allowed them to be in. I rolled back the OS update and the problem was immediately resolved. Unfortunately I didn't keep the OSD logs, but they lead me to this thread from ceph-users: https://www.mail-archive.com/ceph-users@ceph.io/msg18474.html . I wonder if we have an issue with a very recent kernel update. I should be able to reproduce if it's likely to be of use to anybody, but for now I've rolled back this OS update and disabled automatic updating on my other nodes. Matt (*) The complete list of changes: $ rpm-ostree db diff d477f98d52bf707d4282f6835b85bed3d60e305a0cf6eb8effd4db4b89607f05 fc214c16d248686d4cf2bb3050b59c559f091692d7af3b07ef896f1b8ab2f161 ostree diff commit from: d477f98d52bf707d4282f6835b85bed3d60e305a0cf6eb8effd4db4b89607f05 ostree diff commit to: fc214c16d248686d4cf2bb3050b59c559f091692d7af3b07ef896f1b8ab2f161 Upgraded: bash 5.2.9-3.fc37 -> 5.2.15-1.fc37 btrfs-progs 6.0.2-1.fc37 -> 6.1.2-1.fc37 clevis 18-12.fc37 -> 18-14.fc37 clevis-dracut 18-12.fc37 -> 18-14.fc37 clevis-luks 18-12.fc37 -> 18-14.fc37 clevis-systemd 18-12.fc37 -> 18-14.fc37 container-selinux 2:2.193.0-1.fc37 -> 2:2.198.0-1.fc37 containerd 1.6.12-1.fc37 -> 1.6.14-2.fc37 containers-common 4:1-73.fc37 -> 4:1-76.fc37 containers-common-extra 4:1-73.fc37 -> 4:1-76.fc37 coreutils 9.1-6.fc37 -> 9.1-7.fc37 coreutils-common 9.1-6.fc37 -> 9.1-7.fc37 crun 1.7.2-2.fc37 -> 1.7.2-3.fc37 curl 7.85.0-4.fc37 -> 7.85.0-5.fc37 dnsmasq 2.87-3.fc37 -> 2.88-1.fc37 ethtool 2:6.0-1.fc37 -> 2:6.1-1.fc37 fwupd 1.8.8-1.fc37 -> 1.8.9-1.fc37 git-core 2.38.1-1.fc37 -> 2.39.0-1.fc37 grub2-common 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-efi-x64 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-pc 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-pc-modules 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-tools 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-tools-minimal 1:2.06-63.fc37 -> 1:2.06-72.fc37 kernel 6.0.15-300.fc37 -> 6.0.18-300.fc37 kernel-core 6.0.15-300.fc37 -> 6.0.18-300.fc37 kernel-modules 6.0.15-300.fc37 -> 6.0.18-300.fc37 libcurl-minimal 7.85.0-4.fc37 -> 7.85.0-5.fc37 libgpg-error 1.45-2.fc37 -> 1.46-1.fc37 libgusb 0.4.2-1.fc37 -> 0.4.3-1.fc37 libksba 1.6.2-1.fc37 -> 1.6.3-1.fc37 libpcap 14:1.10.1-4.fc37 -> 14:1.10.2-1.fc37 libpwquality 1.4.4-11.fc37 -> 1.4.5-1.fc37 libsmbclient 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 libwbclient 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 moby-engine 20.10.20-1.fc37 -> 20.10.21-1.fc37 ncurses 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 ncurses-base 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 ncurses-libs 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 net-tools 2.0-0.63.20160912git.fc37 -> 2.0-0.64.20160912git.fc37 rpm-ostree 2022.16-2.fc37 -> 2022.19-2.fc37 rpm-ostree-libs 2022.16-2.fc37 -> 2022.19-2.fc37 samba-client-libs 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 samba-common 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 samba-common-libs 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 selinux-policy 37.16-1.fc37 -> 37.17-1.fc37 selinux-policy-targeted 37.16-1.fc37 -> 37.17-1.fc37 tpm2-tss 3.2.0-3.fc37 -> 3.2.1-1.fc37 Removed: cracklib-dicts-2.9.7-30.fc37.x86_64 -- Matthew Booth

2 1

Does cephfs subvolume have commands similar to `rbd perf` to query iops, bandwidth, and latency of rbd image?
by 郑亮 13 Feb '23

13 Feb '23

Hi all, Does cephfs subvolume have commands similar to rbd perf to query iops, bandwidth, and latency of rbd image? `ceph fs perf stats` shows metrics of the client side, not the metrics of the cephfs subvolume. What I want to get is the metrics at the subvolume level like below. [root@smd-exporter-5f87dcb946-wt7tl /]# rbd perf image iostat rbd: waiting for initial image stats NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT pool/csi-vol-5c2d22e3-9195-11ed-aab9-a6732a88c7dd 8/s 0/s 215 KiB/s 0 B/s 14.62 ms 0.00 ns pool/csi-vol-10e31971-9196-11ed-aab9-a6732a88c7dd 5/s 0/s 79 KiB/s 0 B/s 13.18 ms 0.00 ns pool/csi-vol-0f6e0dbe-9196-11ed-aab9-a6732a88c7dd 1/s 0/s 23 KiB/s 0 B/s 20.60 ms 0.00 ns pool/csi-vol-88704e7a-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 12 KiB/s 0 B/s 12.38 ms 0.00 ns pool/csi-vol-a87c21e9-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 14 KiB/s 0 B/s 13.21 ms 0.00 ns pool/csi-vol-b6d040a2-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 8 KiB/s 0 B/s 9.88 ms 0.00 ns pool/csi-vol-efb7c236-6fc9-11ed-aab9-a6732a88c7dd 0/s 0/s 3.2 KiB/s 0 B/s 2.96 ms 0.00 ns NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT pool/csi-vol-5c2d22e3-9195-11ed-aab9-a6732a88c7dd 12/s 0/s 165 KiB/s 0 B/s 26.07 ms 0.00 ns pool/csi-vol-10e31971-9196-11ed-aab9-a6732a88c7dd 4/s 0/s 91 KiB/s 0 B/s 9.83 ms 0.00 ns pool/csi-vol-b6d040a2-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 22 KiB/s 0 B/s 15.25 ms 0.00 ns pool/csi-vol-efb7c236-6fc9-11ed-aab9-a6732a88c7dd 1/s 0/s 38 KiB/s 0 B/s 20.40 ms 0.00 ns pool/csi-vol-88704e7a-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 9.6 KiB/s 0 B/s 12.96 ms 0.00 ns pool/csi-vol-a87c21e9-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 10 KiB/s 0 B/s 3.99 ms 0.00 ns pool/csi-vol-0f6e0dbe-9196-11ed-aab9-a6732a88c7dd 0/s 0/s 2.4 KiB/s 0 B/s 7.33 ms 0.00 ns Best Regards, Liang Zheng

2 1

Any issues with podman 4.2 and Quincy?
by Vladimir Brik 13 Feb '23

13 Feb '23

Has anybody run into issues with Quincy and podman 4.2? 4x podman series are not mentioned in https://docs.ceph.com/en/quincy/cephadm/compatibility/ but podman 3x is no longer available in Alma Linux Vlad

2 1

Health warning - POOL_TARGET_SIZE_BYTES_OVERCOMMITED
by Sake Paulusma 13 Feb '23

13 Feb '23

Hello, I configured a stretched cluster on two datacenters. It's working fine, except this weekend the Raw Capicity exceeded 50% and the error POOL_TARGET_SIZE_BYTES_OVERCOMMITED showed up. The command "ceph df" is showing the correct cluster size, but "ceph osd pool autoscale-status" is showing half of the total Raw Capacity. What could be wrong? ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph status cluster: id: adbe7bb6-5h6d-11ed-8511-004449ede0c health: HEALTH_WARN 1 MDSs report oversized cache 1 subtrees have overcommitted pool target_size_bytes services: mon: 5 daemons, quorum host1,host2,host3,host4,host5 (age 4w) mgr: aqsoel11445.nqamuz(active, since 5w), standbys: host1.wujgas mds: 2/2 daemons up, 2 standby osd: 12 osds: 12 up (since 5w), 12 in (since 9w) data: volumes: 2/2 healthy pools: 5 pools, 193 pgs objects: 17.31M objects, 1.2 TiB usage: 5.0 TiB used, 3.8 TiB / 8.8 TiB avail pgs: 192 active+clean 1 active+clean+scrubbing ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 8.8 TiB 3.8 TiB 5.0 TiB 5.0 TiB 56.83 TOTAL 8.8 TiB 3.8 TiB 5.0 TiB 5.0 TiB 56.83 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 449 KiB 2 1.8 MiB 0 320 GiB cephfs.application-tst.meta 2 16 540 MiB 18.79k 2.1 GiB 0.16 320 GiB cephfs.application-tst.data 3 32 4.4 GiB 8.01k 17 GiB 1.33 320 GiB cephfs.application-acc.meta 4 16 11 GiB 3.54M 45 GiB 3.37 320 GiB cephfs.application-acc.data 5 128 1.2 TiB 13.74M 4.8 TiB 79.46 320 GiB ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 448.5k 4.0 4499G 0.0000 1.0 1 on False cephfs.application-tst.meta 539.8M 4.0 4499G 0.0005 4.0 16 on False cephfs.application-tst.data 4488M 51200M 4.0 4499G 0.0444 1.0 32 on False cephfs.application-acc.meta 11430M 4.0 4499G 0.0099 4.0 16 on False cephfs.application-acc.data 1244G 4.0 4499G 1.1062 1.0000 0.9556 1.0 128 on False ----------------------------------------------------------------------------

2 3

mds damage cannot repair
by Andrej Filipcic 13 Feb '23

13 Feb '23

Hi, there is mds damage on our cluster, version 17.2.5, [ { "damage_type": "backtrace", "id": 2287166658, "ino": 3298564401782, "path": "/hpc/home/euliz/.Xauthority" } ] The recursive repair does not fix it, ...ceph tell mds.0 scrub start /hpc/home/euliz force,repair,recursive mds log: 2023-02-10T07:01:34.012+0100 7f46df3ea700 0 mds.0.cache failed to open ino 0x30001c26a76 err -116/0 2023-02-10T07:01:34.012+0100 7f46df3ea700 0 mds.0.cache open_remote_dentry_finish bad remote dentry [dentry #0x1/hpc/home/euliz/.Xauthority [568,head] auth REMOTE(reg) (dversion lock) pv=0 v=4425667830 ino=(nil) state=1073741824 | ptrwaiter=1 0x5560eb33a780] Any clue how to fix this? or remove the file from namespace? it is not important... Thanks, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail:Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

2 2

Migrate a bucket from replicated pool to ec pool
by Boris Behrens 13 Feb '23

13 Feb '23

Hi, we use rgw as our backup storage, and it basically holds only compressed rbd snapshots. I would love to move these out of the replicated into a ec pool. I've read that I can set a default placement target for a user ( https://docs.ceph.com/en/octopus/radosgw/placement/) What does happen to the existing user data? How do I move the existing data to the new pool? Does it somehow interfere with ongoing data upload (it is one internal user, with 800 buckets which constantly get new data and old data removed)? Cheers Boris ps: Can't wait to see some of you at the cephalocon :) -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 4

[RGW - octopus] too many omapkeys on versioned bucket
by Boris Behrens 13 Feb '23

13 Feb '23

Hi, I have one bucket that showed up with a large omap warning, but the amount of objects in the bucket, does not align with the amount of omap keys. The bucket is sharded to get rid of the "large omapkeys" warning. I've counted all the omapkeys of one bucket and it came up with 33.383.622 (rados -p INDEXPOOL listomapkeys INDEXOBJECT | wc -l) I've checked the amount of actual rados objects and it came up with 17.095.877 (rados -p DATAPOOL ls | grep BUCKETMARKER | wc -l) I've checked the bucket index and it came up with 16.738.482 (radosgw-admin bi list --bucket BUCKET | grep -F '"idx":' | wc -l) I have tried to fix it with radosgw-admin bucket check --check-objects --fix --bucket BUCKET but this did not change anything. Is this a known bug or might there be something else going on. How can I investigate further? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 3

Is ceph bootstrap keyrings in use after bootstrap?
by Zhongzhou Cai 13 Feb '23

13 Feb '23

Hi, I'm on Ceph version 16.2.10, and I found there are a bunch of bootstrap keyrings (i.e., client.bootstrap-<mds|mgr|osd|rbd|rbd-mirror|rgw>) located at /var/lib/ceph/bootstrap-<mds|mgr|osd|rbd|rbd-mirror|rgw>/ceph.keyring after bootstrap. Are they still in use after bootstrap? Is it safe to remove them from host and even from ceph monitor? Thanks, Zhongzhou Cai

2 1