April 2024 - ceph-users - lists.ceph.io

Strange placement groups warnings
by Dmitriy Maximov 10 Apr '24

10 Apr '24

Dear Ceph experts, recently we have upgraded our ceph cluster from octopus (15.2.17) to pacific (16.2.14 and then to 16.2.15). Just after upgrade warnings that all (except device_health_metrics pool) our pools have too many placement groups appeared. This warning looks like generated by autoscaler, but there was no suggestions from it. When I started to investigate situation, I turned bulk flag to true for pools intended to store large amount of data (just to look changes). After that most pools reports that they have too few placement groups, but there is still no suggestions from autoscaler (NEW PG_NUM column is empty) # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK metadata 22567M 3.0 57255G 0.0012 1.0 32 warn False rbd_hdd02_ec_meta 20817k 3.0 365.3T 0.0000 1.0 32 warn False device_health_metrics 1002M 3.0 205.4T 0.0000 1.0 1 on False rbd_ssd_ec_meta 0 3.0 57255G 0.0000 1.0 32 warn False cmd_spool 45094G 1.3333333730697632 205.4T 0.2859 1.0 512 warn True data 60496G 3.0 365.3T 0.4851 1.0 512 warn True rbd 2625G 3.0 205.4T 0.0374 1.0 64 warn True kedr_spool 6900G 10240G 1.3333333730697632 205.4T 0.0649 1.0 128 warn True gcf_spool 1119M 1.3333333730697632 205.4T 0.0000 1.0 32 warn True rbd_ssd 107.8G 3.0 57255G 0.0056 1.0 32 warn True data_ssd 775.8G 3.0 57255G 0.0407 1.0 128 warn True only warnings in 'ceph status' and 'ceph health detail' # ceph health detail HEALTH_WARN 6 pools have too few placement groups; 4 pools have too many placement groups [WRN] POOL_TOO_FEW_PGS: 6 pools have too few placement groups Pool data has 512 placement groups, should have 2048 Pool rbd has 64 placement groups, should have 512 Pool kedr_spool has 128 placement groups, should have 256 Pool gcf_spool has 32 placement groups, should have 256 Pool rbd_ssd has 32 placement groups, should have 2048 Pool data_ssd has 128 placement groups, should have 2048 [WRN] POOL_TOO_MANY_PGS: 4 pools have too many placement groups Pool metadata has 32 placement groups, should have 32 Pool rbd_hdd02_ec_meta has 32 placement groups, should have 32 Pool rbd_ssd_ec_meta has 32 placement groups, should have 32 Pool cmd_spool has 512 placement groups, should have 512 When I changed number of placement groups of pool kedr_spool from 128 to 256 as suggested in warning, data rebalanced successfully, but now pool warns that it have too many placement groups: Pool kedr_spool has 256 placement groups, should have 256. The most strange for me in these warnings is that the current number of PGs is the same that should be. What is it and what how to solve these warnings? Best regards, Dmitriy

1 0

Issue about execute "ceph fs new"
by elite_stu＠163.com 10 Apr '24

10 Apr '24

Everything goes fine except execute "ceph fs new kingcephfs cephfs-king-metadata cephfs-king-data", its shows 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds. But i see there is one mds services running, please help me to fix the issue, thanks a lot. bash-4.4$ bash-4.4$ ceph fs new kingcephfs cephfs-king-metadata cephfs-king-data new fs with metadata pool 7 and data pool 8 bash-4.4$ bash-4.4$ ceph -s cluster: id: de9af3fe-d3b1-4a4b-bf61-929a990295f6 health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 3 daemons, quorum a,b,c (age 2d) mgr: a(active, since 2d), standbys: b mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 2d), 3 in (since 2d) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 2/2 healthy pools: 14 pools, 233 pgs objects: 592 objects, 450 MiB usage: 1.5 GiB used, 208 GiB / 210 GiB avail pgs: 233 active+clean io: client: 921 B/s rd, 1 op/s rd, 0 op/s wr bash-4.4$ bash-4.4$ ceph fs status myfs - 0 clients ==== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active myfs-a Reqs: 0 /s 10 13 12 0 0-s standby-replay myfs-b Evts: 0 /s 0 3 2 0 POOL TYPE USED AVAIL myfs-metadata metadata 180k 65.9G myfs-replicated data 0 65.9G kingcephfs - 0 clients ========== POOL TYPE USED AVAIL cephfs-king-metadata metadata 0 65.9G cephfs-king-data data 0 65.9G MDS version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) bash-4.4$ bash-4.4$ bash-4.4$ bash-4.4$ bash-4.4$ ceph mds stat myfs:1 kingcephfs:0 {myfs:0=myfs-a=up:active} 1 up:standby-replay bash-4.4$ bash-4.4$ ceph fs ls name: myfs, metadata pool: myfs-metadata, data pools: [myfs-replicated ] name: kingcephfs, metadata pool: cephfs-king-metadata, data pools: [cephfs-king-data ] bash-4.4$

2 6

Regarding write on CephFS - Operation not permitted
by elite_stu＠163.com 10 Apr '24

10 Apr '24

Dears , CephFS mount succesful, however i encounted an issue about write file with error message "Operation not permitted". It happend even the file chmod 777, please help me to solve, thanks a lot! [root@vm-04 mycephfs]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/centos-root xfs 46G 1.3G 44G 3% / devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs tmpfs 1.9G 8.5M 1.9G 1% /run tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/vda1 xfs 1014M 145M 870M 15% /boot tmpfs tmpfs 379M 0 379M 0% /run/user/0 192.168.100.7:6789,192.168.100.8:6789,192.168.100.9:6789:/ ceph 19G 0 19G 0% /mnt/mycephfs [root@vm-04 mycephfs]# [root@vm-04 mycephfs]# pwd /mnt/mycephfs [root@vm-04 mycephfs]# [root@vm-04 mycephfs]# ls -rlt total 0 -rw-r--r-- 1 root root 0 Apr 8 20:44 aa.txt -rw-r--r-- 1 root root 0 Apr 8 20:54 bb.txt [root@vm-04 mycephfs]# cp aa.txt cc.txt cp: error reading ‘aa.txt’: Operation not permitted cp: failed to extend ‘cc.txt’: Operation not permitted [root@vm-04 mycephfs]# [root@vm-04 mycephfs]# echo "123" > aa.txt -bash: echo: write error: Operation not permitted [root@vm-04 mycephfs]# [root@vm-04 mycephfs]#

1 1

Ceph alert module different code path?
by Torkil Svensgaard 09 Apr '24

09 Apr '24

Hi We are getting a lot of odd warnings from the alert module by email, alerts which never seem to show up on "ceph -s". Here's a somewhat funny one: Date: Tue, 09 Apr 2024 16:48:06 -0000 " --- New --- [WARN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering pg 11.a0c is stuck peering since forever, current state peering, last acting [504,491,264,535,157,297] " Date: Tue, 09 Apr 2024 16:48:11 -0000 " --- Cleared --- [WARN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering pg 11.a0c is stuck peering since forever, current state peering, last acting [504,491,264,535,157,297] " They often come in pairs like this, first a new (ghost?) warning and then a cleared warning right after. Since these never show up on "ceph -s" I am guessing this a separate code path and the alert module is somewhat broken? Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

1 0

slow pwrite64()s to ceph
by Kelly, Mark (RIS-BCT) 09 Apr '24

09 Apr '24

Hi, Thank you for any time. We are tracking some very slow pwrite64() calls to a ceph filesystem - 20965 11:04:24.049186 <... pwrite64 resumed>) = 65536 <4.489594> 20966 11:04:24.069765 <... pwrite64 resumed>) = 65536 <4.508859> 20967 11:04:24.090354 <... pwrite64 resumed>) = 65536 <4.510256> But other pwrite64()s from the same program in other threads to other files on the same ceph fs seem fine; we cannot really reproduce this, but it happens occasionally. It seems we are spending some time in ceph_aio_write() when this is happening (see call graph below) I've noticed THP (transparent Huge Pages) is enabled. We are running version 15.2.17 on CentOS 7.9 Do not seem to be under any significant memory pressure when this happens, just many threads of this app blocked on i/o in pwrite64()s. I am suggesting an upgrade, but until then, do you think this situation involves ceph and could be improved if we disable THP ? Thanks for any advice or suggestion, -mark Call graph of app when slow pwrites64()s are happening - --87.08%--system_call_fastpath |--58.20%--sys_pwrite64 | --58.03%--vfs_write | do_sync_write | ceph_aio_write | |--54.33%--generic_file_buffered_write | | |--27.09%--ceph_write_begin | | | |--17.01%--grab_cache_page_write_begin | | | | |--7.51%--add_to_page_cache_lru | | | | | |--4.28%--__add_to_page_cache_locked | | | | | | --3.95%--mem_cgroup_cache_charge | | | | | | mem_cgroup_charge_common | | | | | | --3.75%--__mem_cgroup_commit_charge | | | | | --3.23%--lru_cache_add | | | | | __lru_cache_add | | | | | --2.94%--pagevec_lru_move_fn | | | | | |--0.70%--mem_cgroup_page_lruvec | | | | | |--0.67%--__pagevec_lru_add_fn | | | | | --0.57%--release_pages | | | | |--5.31%--__page_cache_alloc | | | | | --5.00%--alloc_pages_current | | | | | __alloc_pages_nodemask | | | | | --4.23%--get_page_from_freelist | | | | | |--1.85%--__rmqueue | | | | | | --1.49%--list_del | | | | | | __list_del_entry | | | | | --1.74%--list_del | | | | | __list_del_entry | | | | --3.92%--__find_lock_page | | | | __find_get_page | | | | --3.46%--radix_tree_lookup_slot | | | | |--2.76%--radix_tree_descend | | | | --0.70%--__radix_tree_lookup | | | | radix_tree_descend | | | --9.45%--ceph_update_writeable_page | | | --8.94%--readpage_nounlock | | | --8.60%--ceph_osdc_readpages | | | --8.18%--submit_request | | | __submit_request | | | calc_target.isra.50 | | | ceph_pg_to_up_acting_osds | | | crush_do_rule | | | crush_choose_firstn | | | |--4.89%--crush_choose_firstn | | | | is_out.isra.2.part.3 | | | --3.30%--crush_bucket_choose | | |--14.50%--copy_user_enhanced_fast_string | | |--6.34%--ceph_write_end | | | |--3.76%--set_page_dirty | | | | ceph_set_page_dirty | | | | --2.86%--__set_page_dirty_nobuffers | | | | |--0.92%--_raw_spin_unlock_irqrestore | | | | --0.59%--radix_tree_tag_set | | | --1.88%--unlock_page | | | __wake_up_bit | | |--3.44%--iov_iter_fault_in_readable | | --2.70%--mark_page_accessed | |--2.54%--mutex_lock | | __mutex_lock_slowpath | | --2.14%--schedule_preempt_disabled | | __schedule | | | | | |--0.82%--finish_task_switch | | | __perf_event_task_sched_in | | | perf_pmu_enable | | | x86_pmu_enable | | | --0.80%--intel_pmu_enable_all | | | --0.74%--__intel_pmu_enable_all.isra.23 | | | --0.69%--native_write_msr_safe | | --0.75%--__perf_event_task_sched_out | --0.56%--mutex_unlock | __mutex_unlock_slowpath ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

1 0

DB/WALL and RGW index on the same NVME
by Lukasz Borek 08 Apr '24

08 Apr '24

Hi! I'm working on a POC cluster setup dedicated to backup app writing objects via s3 (large objects, up to 1TB transferred via multipart upload process). Initial setup is 18 storage nodes (12HDDs + 1 NVME card for DB/WALL) + EC pool. Plan is to use cephadm. I'd like to follow good practice and put the RGW index pool on a no-rotation drive. Question is how to do it? - replace a few HDDs (1 per node) with a SSD (how many? 4-6-8?) - reserve space on NVME drive on each node, create lv based OSD and let rgb index use the same NVME drive as DB/WALL Thoughts? -- Lukasz

4 6

Bucket usage per storage classes
by Ondřej Kukla 08 Apr '24

08 Apr '24

Hello, I’m playing around with Storage classes in rgw and I’m looking for ways to see per bucket statistics for the diferent storage classes (for billing purposes etc.). I though that I would add another object to the bucket usage response like for multiparts - rgw.multimeta, but it’s counted under the rgw.mainIs. Is there some option to get this info? Ondrej Bucket usage I’m referring to "usage": { "rgw.main": { "size": 1333179575, "size_actual": 1333190656, "size_utilized": 1333179575, "size_kb": 1301934, "size_kb_actual": 1301944, "size_kb_utilized": 1301934, "num_objects": 4 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }

3 5

RBD Unmap busy while no "normal" process holds it.
by Nicolas FOURNIL 08 Apr '24

08 Apr '24

Hello, I've got a strange issue with ceph (ceph-adm). We use Incus (LXD reborn) and we used to add/remove containers. We had moved to Ceph for better scalability but sometimes we get this bug : https://discuss.linuxcontainers.org/t/howto-delete-container-with-ceph-rbd-… And I respawn this old bug in Incus : https://discuss.linuxcontainers.org/t/incus-0-x-and-ceph-rbd-map-is-sometim… (thanks for Stephane GRABER help !) After working on it for several hours. I find this : I do a loop who do this script (create image / map it / format it /mount it / write on it / unmount / unmap / delete image) : rbd create image1 --size 1024 --pool customers-clouds.ix-mrs2.fr.eho || exit $? RBD_DEVICE=$(rbd map customers-clouds.ix-mrs2.fr.eho/image1 || exit $? ) mkfs.ext4 ${RBD_DEVICE} || exit $? mount ${RBD_DEVICE} /media/test || exit $? dd if=/dev/zero of=/media/test/test.out sleep 10 rm /media/test/test.out || exit $? umount ${RBD_DEVICE} || exit $? rbd unmap ${RBD_DEVICE} || exit $? rbd rm customers-clouds.ix-mrs2.fr.eho/image1 || exit $? sleep 1 It works for hours without any issue BUT if I add an OSD while doing this loop I get this : + sleep 10 + rm /media/test/test.out + umount /dev/rbd0 + rbd unmap /dev/rbd0 rbd: sysfs write failed rbd: unmap failed: (16) Device or resource busy + exit 16 And ... of course the winner is always the podman of the latest OSD i've added (it holds a copy of the mounting point) root@ceph01:~# grep rbd0 /proc/*/mountinfo/proc/1415299/mountinfo:1959 1837 252:0 / /rootfs/media/test rw,relatime - ext4 /dev/rbd0 rw,stripe=16/proc/1415301/mountinfo:1959 1837 252:0 / /rootfs/media/test rw,relatime - ext4 /dev/rbd0 rw,stripe=16 root@ceph01:~# cat /proc/1415299/cmdline /run/podman-init--/usr/bin/ceph-osd-nosd.26-f--setuserceph--setgroupceph--default-log-to-file=false--default-log-to-journald=true--default-log-to-stderr=falseroot@ceph01-b15r56-2r016.ix-mrs2.fr.eho.admin:~# root@ceph01:~# cat /proc/1415301/cmdline /usr/bin/ceph-osd-nosd.26-f--setuserceph--setgroupceph--default-log-to-file=false--default-log-to-journald=true--default-log-to-stderr=falseroot@ceph01-b15r56-2r016.ix-mrs2.fr.eho.admin:~# Our setup is straight forward : latest stable Ceph release, latest Debian stable release, and deploying via ceph-adm Did someone have such a problem ? Where is the best place to report this bug ?

1 0

NFS never recovers after slow ops
by Torkil Svensgaard 06 Apr '24

06 Apr '24

Hi Cephadm Reef 18.2.1 Started draining 5 18-20 TB HDD OSDs (DB/WAL om NVMe) on one host. Even with osd_max_backfills at 1 the OSDs get slow ops from time to time which seems odd as we recently did a huge reshuffle[1] involving the same host without seeing these slow ops. I guess one difference is the disks were then only getting writes when they were added and now they are only being used for reads as they are being drained. The slow ops eventually go away but I'm seeing stuck nfsd threads from RBD exports lingering on forever. I have to reboot the NFS server to get it going again, restarting nfs-server also just hangs. Here's a stack trace from dmesg: " [Sat Apr 6 17:44:52 2024] INFO: task nfsd:52502 blocked for more than 1245 seconds. [Sat Apr 6 17:44:52 2024] Not tainted 5.14.0-362.8.1.test2.el9_3.x86_64 #1 [Sat Apr 6 17:44:52 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Apr 6 17:44:52 2024] task:nfsd state:D stack:0 pid:52502 ppid:2 flags:0x00004000 [Sat Apr 6 17:44:52 2024] Call Trace: [Sat Apr 6 17:44:52 2024] <TASK> [Sat Apr 6 17:44:52 2024] __schedule+0x20a/0x550 [Sat Apr 6 17:44:52 2024] schedule+0x2d/0x70 [Sat Apr 6 17:44:52 2024] schedule_timeout+0x11f/0x160 [Sat Apr 6 17:44:52 2024] ? xfs_trans_read_buf_map+0x133/0x300 [xfs] [Sat Apr 6 17:44:52 2024] ? xfs_btree_read_buf_block.constprop.0+0x9a/0xd0 [xfs] [Sat Apr 6 17:44:52 2024] __down_common+0x11f/0x200 [Sat Apr 6 17:44:52 2024] ? xfs_btree_read_buf_block.constprop.0+0x30/0xd0 [xfs] [Sat Apr 6 17:44:52 2024] down+0x43/0x60 [Sat Apr 6 17:44:52 2024] xfs_buf_lock+0x2d/0xe0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_buf_find_lock+0x45/0xf0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_buf_lookup.constprop.0+0xe4/0x170 [xfs] [Sat Apr 6 17:44:52 2024] xfs_buf_get_map+0xc1/0x3a0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_buf_read_map+0x54/0x290 [xfs] [Sat Apr 6 17:44:52 2024] ? xfs_imap_to_bp+0x4e/0x70 [xfs] [Sat Apr 6 17:44:52 2024] ? xfs_imap_lookup+0x173/0x1d0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_trans_read_buf_map+0x133/0x300 [xfs] [Sat Apr 6 17:44:52 2024] ? xfs_imap_to_bp+0x4e/0x70 [xfs] [Sat Apr 6 17:44:52 2024] xfs_imap_to_bp+0x4e/0x70 [xfs] [Sat Apr 6 17:44:52 2024] xfs_iget_cache_miss+0xa2/0x370 [xfs] [Sat Apr 6 17:44:52 2024] xfs_iget+0x19f/0x270 [xfs] [Sat Apr 6 17:44:52 2024] ? __pfx_nfsd_acceptable+0x10/0x10 [nfsd] [Sat Apr 6 17:44:52 2024] xfs_nfs_get_inode.isra.0+0x5e/0xa0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_fs_fh_to_dentry+0x48/0xb0 [xfs] [Sat Apr 6 17:44:52 2024] exportfs_decode_fh_raw+0x60/0x2e0 [Sat Apr 6 17:44:52 2024] ? exp_find_key+0x99/0x1e0 [nfsd] [Sat Apr 6 17:44:52 2024] ? rcu_nocb_try_bypass+0x4d/0x440 [Sat Apr 6 17:44:52 2024] ? __kmalloc+0x19b/0x370 [Sat Apr 6 17:44:52 2024] ? __pfx_put_cred_rcu+0x10/0x10 [Sat Apr 6 17:44:52 2024] ? call_rcu+0x114/0x310 [Sat Apr 6 17:44:52 2024] nfsd_set_fh_dentry+0x2b9/0x470 [nfsd] [Sat Apr 6 17:44:52 2024] fh_verify+0x1b3/0x2f0 [nfsd] [Sat Apr 6 17:44:52 2024] nfsd4_putfh+0x3e/0x70 [nfsd] [Sat Apr 6 17:44:52 2024] nfsd4_proc_compound+0x44e/0x700 [nfsd] [Sat Apr 6 17:44:52 2024] nfsd_dispatch+0x53/0x170 [nfsd] [Sat Apr 6 17:44:52 2024] svc_process_common+0x357/0x640 [sunrpc] [Sat Apr 6 17:44:52 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd] [Sat Apr 6 17:44:52 2024] ? __pfx_nfsd+0x10/0x10 [nfsd] [Sat Apr 6 17:44:52 2024] svc_process+0x12d/0x180 [sunrpc] [Sat Apr 6 17:44:52 2024] nfsd+0xd5/0x190 [nfsd] [Sat Apr 6 17:44:52 2024] kthread+0xe0/0x100 [Sat Apr 6 17:44:52 2024] ? __pfx_kthread+0x10/0x10 [Sat Apr 6 17:44:52 2024] ret_from_fork+0x2c/0x50 [Sat Apr 6 17:44:52 2024] </TASK> " Stack: " [root@cogsworth ~]# cat /proc/52502/stack [<0>] xfs_buf_lock+0x2d/0xe0 [xfs] [<0>] xfs_buf_find_lock+0x45/0xf0 [xfs] [<0>] xfs_buf_lookup.constprop.0+0xe4/0x170 [xfs] [<0>] xfs_buf_get_map+0xc1/0x3a0 [xfs] [<0>] xfs_buf_read_map+0x54/0x290 [xfs] [<0>] xfs_trans_read_buf_map+0x133/0x300 [xfs] [<0>] xfs_imap_to_bp+0x4e/0x70 [xfs] [<0>] xfs_iget_cache_miss+0xa2/0x370 [xfs] [<0>] xfs_iget+0x19f/0x270 [xfs] [<0>] xfs_nfs_get_inode.isra.0+0x5e/0xa0 [xfs] [<0>] xfs_fs_fh_to_dentry+0x48/0xb0 [xfs] [<0>] exportfs_decode_fh_raw+0x60/0x2e0 [<0>] nfsd_set_fh_dentry+0x2b9/0x470 [nfsd] [<0>] fh_verify+0x1b3/0x2f0 [nfsd] [<0>] nfsd4_putfh+0x3e/0x70 [nfsd] [<0>] nfsd4_proc_compound+0x44e/0x700 [nfsd] [<0>] nfsd_dispatch+0x53/0x170 [nfsd] [<0>] svc_process_common+0x357/0x640 [sunrpc] [<0>] svc_process+0x12d/0x180 [sunrpc] [<0>] nfsd+0xd5/0x190 [nfsd] [<0>] kthread+0xe0/0x100 [<0>] ret_from_fork+0x2c/0x50 " The nfsd treads do not recover even with nobackfill set, so the cluster is essential idle: " [root@lazy ~]# ceph -s cluster: id: XXXXXXXXXXXXXXXXXXXXXXXXX health: HEALTH_ERR nobackfill,noscrub,nodeep-scrub flag(s) set 1 scrub errors Possible data damage: 1 pg inconsistent 631 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum lazy,jolly,happy,dopey,sleepy (age 2d) mgr: jolly.tpgixt(active, since 3d), standbys: dopey.lxajvk, lazy.xuhetq mds: 1/1 daemons up, 2 standby osd: 537 osds: 537 up (since 8h), 537 in (since 10d); 917 remapped pgs flags nobackfill,noscrub,nodeep-scrub data: volumes: 1/1 healthy pools: 15 pools, 10849 pgs objects: 548.99M objects, 1.1 PiB usage: 1.9 PiB used, 2.3 PiB / 4.2 PiB avail pgs: 97810419/3182590113 objects misplaced (3.073%) 9931 active+clean 893 active+remapped+backfill_wait 24 active+remapped+backfilling 1 active+clean+inconsistent io: client: 3.5 KiB/s rd, 2.0 MiB/s wr, 5 op/s rd, 115 op/s wr " Any ideas on how to get the nfsd threads to recover? There must be stuck ceph I/O somewhere which never time out or something like that? Mvh. Torkil [1] https://www.spinics.net/lists/ceph-users/msg81549.html -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

2 2

question regarding access cephFS from external network.
by elite_stu＠163.com 06 Apr '24

06 Apr '24

Dears , I have a question about cephfs port 6789 is with type of ClusterIP, then how can i access it from external network? [root@vm-01 examples]# kubectl get svc -nrook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE rook-ceph-mgr ClusterIP 10.106.49.230 <none> 9283/TCP 6d23h rook-ceph-mgr-dashboard ClusterIP 10.96.37.100 <none> 8443/TCP 6d23h rook-ceph-mgr-dashboard-external-https NodePort 10.108.78.191 <none> 8443:31082/TCP 6d21h rook-ceph-mon-a ClusterIP 10.107.62.113 <none> 6789/TCP,3300/TCP 3h12m rook-ceph-mon-b ClusterIP 10.103.94.71 <none> 6789/TCP,3300/TCP 3h12m rook-ceph-mon-d ClusterIP 10.110.210.113 <none> 6789/TCP,3300/TCP 3h1m rook-ceph-rgw-my-store ClusterIP 10.98.150.97 <none> 80/TCP 4d6h rook-ceph-rgw-my-store-external NodePort 10.111.249.203 <none> 80:30514/TCP 4d5h

1 0

cephadm: daemon osd.x on yyy is in error state
by Zakhar Kirpichenko 06 Apr '24

06 Apr '24

Hi, A disk failed in our cephadm-managed 16.2.15 cluster, the affected OSD is down, out and stopped with cephadm, I also removed the failed drive from the host's service definition. The cluster has finished recovering but the following warning persists: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.11 on ceph02 is in error state Is it possible to remove or suppress this warning without having to completely remove the OSD? I would appreciate any advice or pointers. Best regards, Zakhar

2 3

"ceph orch daemon add osd" deploys broken OSD
by service.plant＠ya.ru 06 Apr '24

06 Apr '24

Hi everybody. I've faced the situation when I cannot redeploy OSD on a new disk So, I need to replace osd.30 cuz disk always reports about problems with I\O. I do `ceph orch daemon osd.30 --replace` Then I zap DB ``` root@server-2:/# ceph-volume lvm zap /dev/ceph-db/db-88 --> Zapping: /dev/ceph-db/db-88 Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-db/db-88 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0247342 s, 424 MB/s --> Zapping successful for: <LV: /dev/ceph-db/db-88> ``` And now zap DATA ``` root@server-2:/# ceph-volume lvm zap /dev/sdn --> Zapping: /dev/sdn --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdn bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.35239 s, 7.8 MB/s --> Zapping successful for: <Raw Device: /dev/sdn> ``` Okay, now disk is ready and orchestrator confirms it ``` root@server-1:~# ceph orch device ls host server-2 --refresh server-2 /dev/sdn hdd ST18000NM008J_5000c500d80398bf 16.3T Yes 4m ago ``` Now its time for orchestrator to add new osd ``` root@server-1:~# ceph orch daemon add osd server-2:data_devices=/dev/sdn,db_devices=/dev/ceph-db/db-88 Created no osd(s) on host server-2; already created? ``` But it gives osd.30 in state down. If I try to run systemd service manually, it cannot start because ``` Apr 02 12:30:41 server-2 systemd[1]: Started Ceph osd.30 for ea98e312-dfd9-11ee-a226-33f018c3a407. Apr 02 12:30:41 server-2 bash[3316003]: /bin/bash: /var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/osd.30/unit.run: No such file or directory Apr 02 12:30:41 server-2 systemd[1]: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407(a)osd.30.service: Main process exited, code=exited, status=127/n/a Apr 02 12:30:41 server-2 bash[3316014]: /bin/bash: /var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/osd.30/unit.poststop: No such file or directory Apr 02 12:30:41 server-2 systemd[1]: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407(a)osd.30.service: Failed with result 'exit-code'. Apr 02 12:30:51 server-2 systemd[1]: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407(a)osd.30.service: Scheduled restart job, restart counter is at 1. Apr 02 12:30:51 server-2 systemd[1]: Stopped Ceph osd.30 for ea98e312-dfd9-11ee-a226-33f018c3a407. ``` And even if I try to redeploy osd.30 by ceph orch osd redeploy osd.30 then I get the error in ceph -W cephadm ``` 2024-04-02T12:41:39.856767+0000 mgr.server-2.opelxj (mgr.2994187) 5453 : cephadm [INF] Reconfiguring daemon osd.30 on server-2 2024-04-02T12:41:41.048352+0000 mgr.server-2.opelxj (mgr.2994187) 5454 : cephadm [ERR] cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-ea98e312-dfd9-11ee-a226-33f018c3a407-osd-30 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407-osd-30 Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-ea98e312-dfd9-11ee-a226-33f018c3a407-osd.30 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407-osd.30 Reconfig daemon osd.30 ... Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 10700, in <module> File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 10688, in main File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 6620, in command_deploy_from File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 6638, in _common_deploy File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 6666, in _dispatch_deploy File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 3792, in deploy_daemon File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 3078, in create_daemon_dirs File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 708, in write_new IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/osd.30/config.new' -> '/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/osd.30/config' ``` Could anyone point direction where I should dig into? Thanks in advance!

2 2

How to Identify Bottlenecks in RBD job
by Henry lol 05 Apr '24

05 Apr '24

Hello, I executed a long-running rbd flatten job on the client and expected one of the resources, such as CPU, memory, or network I/O, to reach its maximum usage. However, they didn’t. How can I find the bottleneck to improve performance? Specifically, cpu usage of each msgr-worker thread of rbd is under 60%, network usage is under 30% and memory is rarely used.

1 1

Upgraded to Quincy 17.2.7: some S3 buckets inaccessible
by Lorenz Bausch 05 Apr '24

05 Apr '24

Hi everybody, we upgraded our containerized Red Hat Pacific cluster to the latest Quincy release (Community Edition). The upgrade itself went fine, the cluster is HEALTH_OK, all daemons run the upgraded version: ---- %< ---- $ ceph -s cluster: id: 68675a58-cf09-4ebd-949c-b9fcc4f2264e health: HEALTH_OK services: mon: 5 daemons, quorum node02,node03,node04,node05,node01 (age 25h) mgr: node03.ztlair(active, since 25h), standbys: node01.koymku, node04.uvxgvp, node02.znqnhg, node05.iifmpc osd: 408 osds: 408 up (since 22h), 408 in (since 7d) rgw: 19 daemons active (19 hosts, 1 zones) data: pools: 11 pools, 8481 pgs objects: 236.99M objects, 544 TiB usage: 1.6 PiB used, 838 TiB / 2.4 PiB avail pgs: 8385 active+clean 79 active+clean+scrubbing+deep 17 active+clean+scrubbing io: client: 42 MiB/s rd, 439 MiB/s wr, 2.15k op/s rd, 1.64k op/s wr --- $ ceph versions | jq .overall { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 437 } ---- >% ---- After all the daemons were upgraded we started noticing some RGW buckets which are inaccessible. s3cmd failed with NoSuchKey: ---- %< ---- $ s3cmd la -l ERROR: S3 error: 404 (NoSuchKey) ---- >% ---- The buckets still exists according to "radosgw-admin bucket list". Out of the ~600 buckets, 13 buckets are unaccessible at the moment: ---- %< ---- $ radosgw-admin bucket radoslist --tenant xy --uid xy --bucket xy 2024-04-03T12:13:40.607+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed 2024-04-03T12:13:40.609+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed ---- >% ---- The affected buckets are comparatively large, around 4 - 7 TB, but not all buckets of that size are affected. Using "rados -p rgw.buckets.data ls" it seems like all the objects are still there, although "rados -p rgw.buckets.data get objectname -" only prints unusable (?) binary data, even for objects of intact buckets. Overall we're facing around 60 TB of customer data which are just gone at the moment. Is there a way to recover from this situation or further narrowing down the root cause of the problem? Kind regards, Lorenz

3 6

Re: Dashboard and Object Gateway
by Lawson, Nathan 04 Apr '24

04 Apr '24

I am running into this issue as well with a cephadm reef deploy, are there any updates?

1 0

Slow ops during recovery for RGW index pool only when degraded OSD is primary
by Wesley Dillingham 04 Apr '24

04 Apr '24

I am fighting an issue on an 18.2.0 cluster where a restart of an OSD which supports the RGW index pool causes crippling slow ops. If the OSD is marked with primary-affinity of 0 prior to the OSD restart no slow ops are observed. If the OSD has a primary affinity of 1 slow ops occur. The slow ops only occur during the recovery period of the OMAP data and further only occur when client activity is allowed to pass to the cluster. Luckily I am able to test this during periods when I can disable all client activity at the upstream proxy. Given the behavior of the primary affinity changes preventing the slow ops I think this may be a case of recovery being more detrimental than backfill. I am thinking that causing an pg_temp acting set by forcing backfill may be the right method to mitigate the issue. [1] I believe that reducing the PG log entries for these OSDs would accomplish that but I am also thinking a tuning of osd_async_recovery_min_cost [2] may also accomplish something similar. Not sure the appropriate tuning for that config at this point or if there may be a better approach. Seeking any input here. Further if this issue sounds familiar or sounds like another condition within the OSD may be at hand I would be interested in hearing your input or thoughts. Thanks! [1] https://docs.ceph.com/en/latest/dev/peering/#concepts [2] https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval… Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes(a)wesdillingham.com

3 5

question about rbd_read_from_replica_policy
by Noah Elias Feldt 04 Apr '24

04 Apr '24

Hello, I have a question about a setting for RBD. How exactly does "rbd_read_from_replica_policy" with the value "localize" work? According to the RBD documentation, read operations will be sent to the closest OSD as determined by the CRUSH map. How does the client know exactly which OSD I am closest to? The client is not in the CRUSH map. I can't find much more information about it. How does this work? Thanks noah feldt infrastrutur _____ [Inline image OWAPstImg850138] Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp Tel.: 05772 / 293-100 Fax: 05772 / 293-333 support(a)mittwald.de<mailto:support@mittwald.de> https://www.mittwald.de<https://www.mittwald.de/> Geschäftsführer: Robert Meyer, Florian Jürgens USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds<http://www.mittwald.de/ds> abrufbar. ________________________________

3 2

RBD image metric
by Szabo, Istvan (Agoda) 04 Apr '24

04 Apr '24

Hi, Trying to pull out some metrics from ceph about the rbd images sizes but haven't found anything only pool related metrics. Wonder is there any metric about images or I need to create by myself to collect it with some third party tool? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

CEPHADM_HOST_CHECK_FAILED
by arnoud＠fuga.cloud 04 Apr '24

04 Apr '24

Hi, I’ve added some new nodes to our Ceph cluster. Only did the host add, had not added the OSD’s yet. Due to a configuration error I had to reinstall some of them. But I forgot to remove the nodes from Ceph first. I did a “ceph orch host rm peon7 --offline —force” before re-adding them to the cluster. All the nodes are showing up in the host list (all the peons are the new ones): # ceph orch host ls HOST ADDR LABELS STATUS ceph1 10.103.0.71 ceph2 10.103.0.72 ceph3 10.103.0.73 ceph4 10.103.0.74 compute1 10.103.0.11 compute2 10.103.0.12 compute3 10.103.0.13 compute4 10.103.0.14 controller1 10.103.0.8 controller2 10.103.0.9 controller3 10.103.0.10 peon1 10.103.0.41 peon2 10.103.0.42 peon3 10.103.0.43 peon4 10.103.0.44 peon5 10.103.0.45 peon6 10.103.0.46 peon7 10.103.0.47 peon8 10.103.0.48 peon9 10.103.0.49 peon10 10.103.0.50 peon12 10.103.0.52 peon13 10.103.0.53 peon14 10.103.0.54 peon15 10.103.0.55 peon16 10.103.0.56 But Ceph status still shows an error, which I can’t seem to get rid off. [WRN] CEPHADM_HOST_CHECK_FAILED: 2 hosts fail cephadm check host peon7 (10.103.0.47) failed check: Can't communicate with remote host `10.103.0.47`, possibly because python3 is not installed there or you are missing NOPASSWD in sudoers. [Errno 113] Connect call failed ('10.103.0.47', 22) host peon12 (10.103.0.52) failed check: Can't communicate with remote host `10.103.0.52`, possibly because python3 is not installed there or you are missing NOPASSWD in sudoers. [Errno 113] Connect call failed ('10.103.0.52', 22) [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'peon7' Module 'cephadm' has failed: ‘peon7' From the mgr log: Apr 04 08:33:46 controller2 bash[4031857]: debug 2024-04-04T08:33:46.876+0000 7f2bb5710700 -1 mgr.server reply reply (5) Input/output error Module 'cephadm' has experienced an error and cannot handle commands: 'peon7' Any idea how to clear this error? # ceph --version ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Regards, Arnoud de Jonge.

2 1

RGW services crashing randomly with same message
by Reid Guyett 03 Apr '24

03 Apr '24

Hello, We are currently experiencing a lot of rgw service crashes that all seem to terminate with the same message. We have kept our RGW services at 17.2.5 but the rest of the cluster is 17.2.7 due to a bug introduced in 17.2.7. terminate called after throwing an instance of > 'ceph::buffer::v15_2_0::end_of_buffer' > what(): End of buffer > *** Caught signal (Aborted) ** > in thread 7fcfd1772700 thread_name:radosgw > ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy > (stable) > What would be the best way to figure out what is causing the error? The rest of the crash is quite long in our logs with 10k lines of history. Thanks, Reid

1 0

cephadm auto disk preparation and OSD installation incomplete
by Kuhring, Mathias 03 Apr '24

03 Apr '24

Dear ceph community, We have trouble with new disks not being properly prepared resp. OSDs not being fully installed by cephadm. We just added one new node each with ~40 HDDs each to two of our ceph clusters. In one cluster all but 5 disks got installed automatically. In the other none got installed. We are on ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both clusters. (I haven't added new disks since the last upgrade if I recall correctly). This is our OSD service definition: ``` 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: unmanaged service_name: osd.unmanaged unmanaged: true spec: filter_logic: AND objectstore: bluestore ``` Usually, new disks are installed properly (as expected due to all-available-devices). This time, I can see that LVs were created (via `lsblk`, `lvs`, `cephadm ceph-volume lvm list`). And OSDs are entered to the crushmap. However, they are not assigned to a host yet, nor do they have a type or weight, e.g.: ``` 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd" 518 0 osd.518 down 0 1.00000 519 0 osd.519 down 0 1.00000 520 0 osd.520 down 0 1.00000 521 0 osd.521 down 0 1.00000 522 0 osd.522 down 0 1.00000 ``` And there is also no OSD daemon created (no docker container). So, OSD creation is somehow stuck halfway. I thought of fully cleaning up the OSD/disks. Hopping cephadm might pick them up properly next time. Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap --destroy /dev/sdab` results in these errors: ``` /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing initialization failed: Device or resource busy /usr/bin/docker: stderr --> failed to wipefs device, will try again to workaround probable race condition ``` So, I cleaned up more manually with purging them from crush and "resetting" disk and LV with dd and dmsetup, resp.: ``` ceph osd purge 480 --force dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f513--4513--86c1--c7ad63e6c142 ``` ceph-volume still reported the old volumes, but then zapping actually got rid of them (only cleaned out the left-over entries, I guess). Now, cephadm was able to get one OSD up, when I did this cleanup for only one disk. When I did it in bulk for the rest, they all got stuck again the same way. Looking into ceph-volume logs (here for osd.522 as representative): ``` 0|0[root@ceph-2-11 /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316 -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316 -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316 ``` ceph-volume only reports keyring creation: ``` [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating keyring file for osd.522 [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring --create-keyring --name osd.522 --add-key AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ== ``` In the OSD logs I found a couple of these, but don't know if they are related: ``` 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) ``` Has anyone seen this behaviour before? Or could tell me where I should look next to troubleshoot this (which logs)? Any help is appreciated. Best Wishes, Mathias

2 4

quincy-> reef upgrade non-cephadm
by Christopher Durham 03 Apr '24

03 Apr '24

Hi, I am upgrading my test cluster from 17.2.6 (quincy) to 18.2.2 (reef). As it was an rpm install, i am following the directions here: Reef — Ceph Documentation | | | | Reef — Ceph Documentation | | | The upgrade worked, but I have some observations and questions before I move to my production cluster: 1. I see no systemd units with the fsid in them, as described in the document above. Both before and after the upgrade, my mon and other units are: ceph-mon@<server>.serviceceph-osd(a)[N].service etc Should I be concerned? 2. Does order matter? Based on past upgrades, I do not think so, but I wanted to be sure. For example, can I update: mon/mds/radosgw/mgrs first, then afterwards update the osds? This is what i have done in previous updates and and all was well. 3. Again on order, if a server serves say, a mon and mds, I can't really easily update one without the other, based on shared libraries and such. It appears that that is ok, based on my test cluster, but wanted to be sure. Again if an mds is one of the servers to update, I know I have to updatethe remaining one after max_mds is set to 1 and others are stopped, first. 4. After upgrade of my mgr node I get: "Module [several module names] has missing NOTIFY_TYPES member" in ceph-mgr.<server>.log But the mgr starts up eventually The system is Rocky Linux 8.9 Thanks for any thoughts -Chris

2 1

ceph orchestrator for osds
by Jeffrey Turmelle 03 Apr '24

03 Apr '24

Running on Octopus: While attempting to install a bunch of new OSDs on multiple hosts, I ran some ceph orchestrator commands to install them, such as ceph orch apply osd --all-available-devices ceph orch apply osd -I HDD_drive_group.yaml I assumed these were just helper processes, and they would be short-lived. In fact, they didn’t actually work and I ended up installing each drive by hand like this: ceph orch daemon add osd ceph4.iri.columbia.edu:/dev/sdag However, now I have these services running: # ceph orch ls --service-type=osd NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID osd.HDD_drive_group 2/2 7m ago 6w ceph[456].iri.columbia.edu docker.io/ceph/ceph:v15 2cf504fded39 osd.None 54/0 7m ago - <unmanaged> docker.io/ceph/ceph:v15 2cf504fded39 osd.all-available-devices 1/0 7m ago - <unmanaged> docker.io/ceph/ceph:v15 2cf504fded39 I’m certain none of these actually created any of my running OSD daemons, but I’m not sure if it’s ok to remove them. For example: ceph orch daemon rm osd.all-available-devices ceph orch daemon rm osd.HDD_drive_group ceph orch daemon rm osd.None Does anyone have any insight to this? I can just leave them there, they don’t seem to be doing anything, but on the other hand, I don’t want any new devices to be automatically loaded or any other unintended consequences of these. Thanks for any guidance, Jeff Turmelle International Research Institute for Climate & Society <https://iri.columbia.edu/> The Climate School <https://climate.columbia.edu/> at Columbia University <https://columbia.edu/>

2 1

Re: Multi-MDS
by quaglio＠bol.com.br 03 Apr '24

03 Apr '24

1 0

IO500 CFS ISC 2024
by IO500 Committee 03 Apr '24

03 Apr '24

Call for Submission Stabilization Period: Monday, April 1st - Friday, April 15th, 2024 Submission Deadline: Tuesday, May 3rd, 2024 AoE The IO500 is now accepting and encouraging submissions for the upcoming 14th semi-annual IO500 Production and Research lists, in conjunction with ISC24. Once again, we are also accepting submissions to both the Production and Research 10 Client Node Challenges to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Webpage. The new ranked lists will be announced at the BoF [1]. We hope to see many new results. Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: - Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance and the documentation of how the numbers are achieved. - Support small to extreme-scale Research and Production HPC systems using flexible storage APIs - Maximizing simplicity in running the benchmark suite while offering tunable parameters Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes runs with highly prescribed parameters in an attempt to determine a lower performance bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported Storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. The goals of the community are also multi-fold: 1. Gather historical data for the sake of analysis and to aid predictions of storage futures 2. Collect tuning information to share valuable performance optimizations across the community 3. Encourage vendors and designers to optimize for workloads beyond "hero runs" 4. Establish bounded expectations for users, procurers, and administrators 5. Understand and be able to reproduce performance on storage systems Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. The IO500 follows a two-staged approach. First, there will be a two-week stabilization period during which we encourage the community to verify that the benchmark runs properly on a variety of storage systems. During this period the benchmark may be updated based upon feedback from the community. The final benchmark will then be released. We expect that runs compliant with the rules made during the stabilization period will be valid as a final submission unless a significant defect is found. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather We encourage you to submit [2] to join our community, and to attend the ISC’24 BoF [1] on Tuesday, May 12, 2024 at 10:05am - 11:05am CEST, where we will announce the new IO500 Production and Research lists and their 10 Client Node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. We will also highlight new and interesting results with invited talk(s) at the BoF. [1] https://io500.org/pages/bof-isc24 [2] https://io500.org/submission [3] https://io500.org/rules-submission

1 0

ceph status not showing correct monitor services
by Adiga, Anantha 03 Apr '24

03 Apr '24

Hi Why is "ceph -s" showing only two monitors while three monitor services are running ? # ceph versions { "mon": { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 }, "mgr": { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 }, "osd": { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 36 }, "mds": { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1 }, "rgw": { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 }, "overall": { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 44 } } # ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT crash 3/3 8m ago 2y label:ceph ingress.nfs.nfs 10.45.128.8:2049,9049 4/4 8m ago 2y count:2 mds.cephfs 3/3 8m ago 2y count:3;label:mdss mgr 3/3 8m ago 23M a001s016;a001s017;a001s018;count:3 mon 3/3 8m ago 16h a001s016;a001s017;a001s018;count:3 <== [ 3 monitor services running] nfs.nfs ?:12049 3/3 8m ago 2y a001s016;a001s017;a001s018;count:3 node-exporter ?:9100 3/3 8m ago 2y * osd.unmanaged 36/36 8m ago - <unmanaged> prometheus ?:9095 1/1 10s ago 23M count:1 rgw.ceph ?:8080 3/3 8m ago 19h count-per-host:1;label:rgws root@a001s017:~# ceph -s cluster: id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 health: HEALTH_OK services: mon: 2 daemons, quorum a001s018,a001s017 (age 16h) <== [ shows ONLY 2 monitors running] mgr: a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 54s), 36 in (since 2y) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 43 pools, 1633 pgs objects: 51.81M objects, 77 TiB usage: 120 TiB used, 131 TiB / 252 TiB avail pgs: 1631 active+clean 2 active+clean+scrubbing+deep io: client: 220 MiB/s rd, 448 MiB/s wr, 251 op/s rd, 497 op/s wr # ceph orch ls --service_name=mon NAME PORTS RUNNING REFRESHED AGE PLACEMENT mon 3/3 8m ago 16h a001s016;a001s017;a001s018;count:3 <== [ 3 monitors running ] # ceph orch ps --daemon_type=mon NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (19h) 9m ago 19h 706M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (16h) 66s ago 19h 949M 2048M 16.2.5 6e73176320aa e5e5cb6c256c <== [ 3 mon daemons running ] mon.a001s018 a001s018 running (5w) 2m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 a001s016# systemctl --type=service | grep @mon ceph-604d56db-2fab-45db-a9ea-c418f9a8cca8(a)mon.a001s016.service<mailto:ceph-604d56db-2fab-45db-a9ea-c418f9a8cca8@mon.a001s016.service> loaded active running Ceph mon.a001s016 for 604d56db-2fab-45db-a9ea-c418f9a8cca8 a001s017# systemctl --type=service | grep @mon ceph-604d56db-2fab-45db-a9ea-c418f9a8cca8(a)mon.a001s017.service<mailto:ceph-604d56db-2fab-45db-a9ea-c418f9a8cca8@mon.a001s017.service> loaded active running Ceph mon.a001s017 for 604d56db-2fab-45db-a9ea-c418f9a8cca8 a001s018# systemctl --type=service | grep @mon ceph-604d56db-2fab-45db-a9ea-c418f9a8cca8(a)mon.a001s018.service<mailto:ceph-604d56db-2fab-45db-a9ea-c418f9a8cca8@mon.a001s018.service> loaded active running Ceph mon.a001s018 for 604d56db-2fab-45db-a9ea-c418f9a8cca8 Thank you, Anantha

3 14

ceph and raid 1 replication
by Roberto Maggi ＠ Debian 03 Apr '24

03 Apr '24

Hi every one, I'm new to ceph and I'm still studying it. In my company we decided to test ceph for possible further implementations. Although I undestood its capabilities I'm still doubtful about how to setup replication. Once implemented in production I can accept a little lacking of performance in favor of stability and night sleep, hence, if in testing area I can introduce ceph as network storage, I'd like to replicate some osds' drives as I'd do with raid 1 once in production. The goal would be hosting data for kubernetes storage classes. so questions are: 1) what do you think about this kind of solution 2) How can I setup full replication between osds? thanks in advance Rob

2 2

put bucket notification configuration - access denied
by Giada Malatesta 03 Apr '24

03 Apr '24

Hello everyone, we are facing a problem regarding the s3 operation put bucket notification configuration. We are using Ceph version 17.2.6. We are trying to configure buckets in our cluster so that a notification message is sent via amqps protocol when the content of the bucket change. To do so, we created a local rgw user with "special" capabilities and we wrote ad hoc policies for this user (list of all buckets, read access to all buckets and possibility to add, list and delete bucket configurations). The problems regards the configurations of all buckets except the one he owns, when doing this put bucket notification configuration cross-account operation we get an access denied error. I have the suspect that this problem is related to the version we are using, because when we were doing tests on another cluster we were using version 18.2.1 and we did not face this problem. Can you confirm my hypothesis? Thanks, GM.

2 1

rgw s3 bucket policies limitations (on users)
by garcetto 03 Apr '24

03 Apr '24

good afternoon, i am trying to set bucket policies to allow to different users to access same bucket with different permissions, BUT it seems that is not yet supported, am i wrong? https://docs.ceph.com/en/reef/radosgw/bucketpolicy/#limitations "We do not yet support setting policies on users, groups, or roles." thank you.

2 1

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

OSD: failed decoding part header ERRORS
by Mark Selby 03 Apr '24

03 Apr '24

We have a ceph cluster of only nvme drives. Very recently our overall OSD write latency increase pretty dramatically and our overall thoughput has really decreased. One thing that seems to correlate with the start of this problem are the below ERROR line from the logs. All our OSD nodes are creating these log lines now. Can anyone tell me what this might be telling us? All and any help is greatly appreciated. Mar 31 23:21:56 ceph1d03 ceph-8797e570-96be-11ed-b022-506b4b7d76e1-osd-46[12898]: debug 2024-04-01T03:21:56.953+0000 7effbba51700 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/cls/fifo/cls_fifo.cc:112: ERROR: int rados::cls::fifo::{anonymous}::read_part_header(cls_method_context_t, rados::cls::fifo::part_header*): failed decoding part header -- Mark Selby Sr Linux Administrator, The Voleon Group mselby(a)voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/.

1 0

Failed adding back a node
by Alex 02 Apr '24

02 Apr '24

Hello. We're rebuilding our OSD nodes. Once cluster worked without any issues, this one is being stubborn I attempted to add one back to the cluster and seeing the error below in out logs: cephadm ['--image', 'registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160', 'pull'] 2024-03-27 19:30:53,901 7f49792ed740 DEBUG /bin/podman: 4.6.1 2024-03-27 19:30:53,905 7f49792ed740 INFO Pulling container image registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160... 2024-03-27 19:30:54,045 7f49792ed740 DEBUG /bin/podman: Trying to pull registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160... 2024-03-27 19:30:54,266 7f49792ed740 DEBUG /bin/podman: Error: initializing source docker://registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160: reading manifest 16.2.10-160 in registry.redhat.io/rhceph/rhceph-5-rhel8: manifest unknown 2024-03-27 19:30:54,270 7f49792ed740 INFO Non-zero exit code 125 from /bin/podman pull registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160 2024-03-27 19:30:54,270 7f49792ed740 INFO /bin/podman: stderr Trying to pull registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160... 2024-03-27 19:30:54,270 7f49792ed740 INFO /bin/podman: stderr Error: initializing source docker://registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160: reading manifest 16.2.10-160 in registry.redhat.io/rhceph/rhceph-5-rhel8: manifest unknown 2024-03-27 19:30:54,270 7f49792ed740 ERROR ERROR: Failed command: /bin/podman pull registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160 $ ceph versions { "mon": { "ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)": 1, "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 2 }, "mgr": { "ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)": 1, "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 2 }, "osd": { "ceph version 16.2.10-160.el8cp (6977980612de1db28e41e0a90ff779627cde7a8c) pacific (stable)": 160 }, "mds": {}, "rgw": { "ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)": 3 }, "overall": { "ceph version 16.2.10-160.el8cp (6977980612de1db28e41e0a90ff779627cde7a8c) pacific (stable)": 160, "ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)": 5, "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 4 } } I don't understand why it's trying to pull 16.2.10-160 which doesn't exist. registry.redhat.io/rhceph/rhceph-5-dashboard-rhel8 5 93b3137e7a65 11 months ago 696 MB registry.redhat.io/rhceph/rhceph-5-rhel8 5-416 838cea16e15c 11 months ago 1.02 GB registry.redhat.io/openshift4/ose-prometheus v4.6 ec2d358ca73c 17 months ago 397 MB This happens using cepadm-ansible as well as $ ceph orch ls --export --service_name xxx > xxx.yml $ sudo ceph orch apply -i xxx.yml I tried ceph orch daemon add osd host:/dev/sda which surprisingly created a volume on host:/dev/sda and created an osd i can see in $ ceph osd tree but It did not get added to host I suspect because of the same Podman error and now I'm unable remove it. $ ceph orch osd rm does not work even with the --force flag. I stopped the removal with $ ceph orch osd rm stop after 10+ minutes I'm considering running $ ceph osd purge osd# --force but worried it may only make things worse. ceph -s shows that osd but not up or in. Thanks, and looking forward to any advice!

2 6

Pacific Bug?
by Alex 02 Apr '24

02 Apr '24

Hello Ceph Gurus! I'm running Ceph Pacific version. if I run ceph orch host ls --label osds shows all hosts label osds or ceph orch host ls --host-pattern host1 shows just host1 it works as expected But combining the two the label tag seems to "take over" ceph orch host ls --label osds --host-pattern host1 6 hosts in cluster who had label osds whose hostname matched host1 shows all host with the label osds instead of only host1. So at first the flags seem to act like an OR instead of an AND. ceph orch host ls --label osds --host-pattern foo 6 hosts in cluster who had label osds whose hostname matched foo even though "foo" doesn't even exist ceph orch host ls --label bar --host-pattern host1 0 hosts in cluster who had label bar whose hostname matched host1 if the label and host combo was an OR this should have worked there is no label bar but host1 exists so it just disregards the host-pattern. This started because the osd deployment task had both label and host_pattern. The cluster was attempting to deploy OSDS on all the servers with the given tag instead of the one host we needed, which caused it to go into warning state. If I ran ceph orch ls --export --service_name host1 it also showed both tags and host_pattern. unmanaged: false placement: host_pattern: label: The issue persisted until I removed the label tag. Thanks.

3 5

cephadm shell version not consistent across monitors
by J-P Methot 02 Apr '24

02 Apr '24

Hi, We are still running ceph Pacific with cephadm and we have run into a peculiar issue. When we run the `cephadm shell` command on monitor1, the container we get runs ceph 16.2.9. However, when we run the same command on monitor2, the container runs 16.2.15, which is the current version of the cluster. Why does it do that and is there a way to force it to 16.2.15 on monitor1? Please note that both monitors have the same configuration. Cephadm has been pulled from GitHub for both monitors instead of the package manager's version. -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.

2 1

Replace block drives of combined NVME+HDD OSDs
by Zakhar Kirpichenko 02 Apr '24

02 Apr '24

Hi, Unfortunately, some of our HDDs failed and we need to replace these drives which are parts of "combined" OSDs (DB/WAL on NVME, block storage on HDD). All OSDs are defined with a service definition similar to this one: ``` service_type: osd service_id: ceph02_combined_osd service_name: osd.ceph02_combined_osd placement: hosts: - ceph02 spec: data_devices: paths: - /dev/sda - /dev/sdb - /dev/sdc - /dev/sdd - /dev/sde - /dev/sdf - /dev/sdg - /dev/sdh - /dev/sdi db_devices: paths: - /dev/nvme0n1 - /dev/nvme1n1 filter_logic: AND objectstore: bluestore ``` In the above example, HDDs `sda` and `sdb` are not readable and data cannot be copied over to new HDDs. NVME partitions of `nvme0n1` with DB/WAL data are intact, but I guess that data is useless. I think the best approach is to replace the dead drives and completely rebuild each affected OSD. How should we go about this, preferably in a way that other OSDs on the node remain unaffected and operational? I would appreciate any advice or pointers to the relevant documentation. Best regards, Zakhar

2 3

Questions about rbd flatten command
by Henry lol 02 Apr '24

02 Apr '24

Hello, I executed multiple 'rbd flatten' commands simultaneously on a client. The elapsed time of each flatten job increased as the number of jobs increased, and network I/O was nearly full. so, I have two questions. 1. isn’t the flatten job running within the ceph cluster? Why is client-side network I/O so high? 2. How can I apply qos for each flatten job to reduce network I/O? Sincerely,

2 3

CEPH Quincy installation with multipathd enabled
by youssef.khristo＠elsewedyintel.com 02 Apr '24

02 Apr '24

Greetings community, we have a setup comprising of 6 servers hosting CentOS 8 Minimal Installation with CEPH Quincy version 18.2.2 supported by 20Gbps fiber optics NICs and a dual Xeon Intel processors, bootstrapped the installation on the first node then expanded to the others using the cephadm method, having the monitor services deployed on 5 of these nodes as well as 3 manager nodes. Each server has an NVMe boot disk as well as a 1TBs SATA SSD over which the OSDs are deployed. An EC profile was created with k=3 and m=3, serving a CephFS filesystem on top with NFS exports to serve other servers. Up to this point, the setup is quite stable in the sense that upon emergency reboot or network connection failure the OSDs did not fail and remain functional/started normally after reboot. At a certain point in our project, we had the need to activate the multipathd service, adding the boot drive partition and the CEPH SSD to its blacklist as to not be initialized for use by an mpath partition, the blacklist goes like so: boot blacklist: =============== blacklist { wwid "eui.<drive_id>" } SATA SSD blacklist: =================== blacklist { wwid "naa.<drive_id>" } The above blacklist configuration ensures that both the boot disk as well as CEPH's OSD function properly, with the following being lsblk output: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 894.3G 0 disk └─ceph--<id>-osd--block--<block_id> 252:3 0 894.3G 0 lvm nvme0n1 259:0 0 238.5G 0 disk ├─nvme0n1p1 259:1 0 600M 0 part /boot/efi ├─nvme0n1p2 259:2 0 1G 0 part /boot └─nvme0n1p3 259:3 0 236.9G 0 part ├─centos-root 252:0 0 170G 0 lvm / ├─centos-swap 252:1 0 23.4G 0 lvm [SWAP] ├─centos-var_log_audit 252:2 0 7.5G 0 lvm /var/log/audit ├─centos-home 252:4 0 26G 0 lvm /home └─centos-var_log 252:5 0 10G 0 lvm /var/log In addition to the above multipathd configuration, we have use_devicesfile=1 in /etc/lvm/lvm.conf, with /etc/lvm/devices/system.devices file being like so, with PVID used from the output of the pvdisplay command, and the IDNAME value extracted from the ouput of "ls -lha /dev/disk/by-id": VERSION=1.1.1 IDTYPE=sys_wwid IDNAME=eui.<drive_id> DEVNAME=/dev/nvme0n1p3 PVID=<pvid> PART=3 IDTYPE=sys_wwid IDNAME=naa.<drive_id> DEVNAME=/dev/sda PVID=<pvid> Issues started when performing certain tests regarding the system's integrity, most important of which is emergency shutdown's and reboot of all the nodes, the behavior that follows is that the OSDs are not started automatically as well as their respective LVM volumes not properly showing (except on a single node for some reason), hence the lsblk ouput changes like the snippet below, requiring us rebooting the nodes one by one until all the OSDs are back online: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 894.3G 0 disk nvme0n1 259:0 0 238.5G 0 disk ├─nvme0n1p1 259:1 0 600M 0 part /boot/efi ├─nvme0n1p2 259:2 0 1G 0 part /boot └─nvme0n1p3 259:3 0 236.9G 0 part ├─centos-root 252:0 0 170G 0 lvm / ├─centos-swap 252:1 0 23.4G 0 lvm [SWAP] ├─centos-var_log_audit 252:2 0 7.5G 0 lvm /var/log/audit ├─centos-home 252:4 0 26G 0 lvm /home └─centos-var_log 252:5 0 10G 0 lvm /var/log Without the LVM configuration and the multipathd service enabled, everything works fine, this behavior started happening after the changes. Attempting to manually restart the OSDs from a manager node using ceph orch daemon restart osd.n results in an error state, and even when manually starting the OSD on each node via the bash /var/lib/ceph/<fsid>/osd.0/unit.run we receive the following error: --> Failed to activate via raw: did not find any matching OSD to activate Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-<id>/osd-block-<block_id> --path /var/lib/ceph/osd/ceph-0 --no-mon-config stderr: failed to read label for /dev/ceph-<id>/osd-block-<block_id>: (2) No such file or directory 2024-03-30T12:42:54.014+0000 7f845296a980 -1 bluestore(/dev/ceph-<id>/osd-block-<block_id>) _read_bdev_label failed to open /dev/ceph-<id>/osd-block-<block_id>: (2) No such file or directory --> Failed to activate via LVM: command returned non-zero exit status: 1 --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' --> Failed to activate any OSD(s) A successful run of the same command but with success results in the following output: /bin/bash /var/lib/ceph/<fsid>/osd.0/unit.run Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/ceph--<id>-osd--block--<block_id> Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--<id>-osd--block--<block_id> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5 Running command: /usr/bin/ln -s /dev/mapper/ceph--<id>-osd--block--<block_id> /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 --> ceph-volume raw activate successful for osd ID: 0 ceph-<fsid>-osd-0 4361e2f166bcdeee6e9020dcbb153d3d7eec04e71d5b0b250440d4a3a0833f2c It seems to us as if the logical volume in cases of failure is not even detected at boot by the device-mapper, which is weird, it's also not showing in the output of the dmsetup ls command in cases of failure. What could we be missing here? What seems to be the conflict between CEPH OSDs and the multipathd service or even the LVM configuration? Should the system.devices entry be different than what we set? Is the multipathd blacklisting configuration missing something? We have been working on trial and error experiments for more than a week now and looked at the lvm2 as well as multipathd logs (we could provide them upon request) but to no avail as nothing indicates any errors, just normal logs with the difference being the missing CEPH OSD LVM volume. Best regards

1 0

cephfs inode backtrace information
by Dietmar Rieder 02 Apr '24

02 Apr '24

Hello, I have a question regarding the default pool of a cephfs. According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information? Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data). Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool? Thanks Dietmar

6 10

Drained A Single Node Host On Accident
by adam.ther 02 Apr '24

02 Apr '24

Hello, I have a single node host with a VM as a backup MON,MGR,ect. This has caused all OSD's to be pending as 'deleting', can i safely cancel this deletion request? Regards, Adam

2 1

S3 Partial Reads from Erasure Pool
by E3GH75＠gmail.com 02 Apr '24

02 Apr '24

I am dealing with a cluster that is having terrible performance with partial reads from an erasure coded pool. Warp tests and s3bench tests result in acceptable performance but when the application hits the data, performance plummets. Can anyone clear this up for me, When radosgw gets a partial read does it have to assemble all the rados objects that make up the s3 object before returning the range? With a replicated poll i am seeing 6 to 7 GiB/s of read performance and only 1GiB/s of read from the erasure coded pool which leads me to believe that the replicated pool is returning just the rados objects for the partial s3 object and the erasure coded pool is not.

2 1

Can setting mds_session_blocklist_on_timeout to false minize the session eviction?
by Yongseok Oh 01 Apr '24

01 Apr '24

Hi, CephFS is provided as a shared file system service in a private cloud environment of our company, LINE. The number of sessions is approximately more than 5,000, and session evictions occur several times a day. When session eviction occurs, the message 'Cannot send after transport endpoint shutdown' or 'Permission denied' is displayed and file system access is not possible. Our users are very uncomfortable with this issue. In particular, there are no special problems such as network connection or CPU usage. When I access the machine and take a close look, there are no special problems. After this, users feel the inconvenience of having to perform umount/mount tasks and run the application again. In a Kubernetes environment, recovery is a bit more complicated, which causes a lot of frustration. I tested that by setting the mds_session_blocklist_on_timeout and mds_session_blocklist_on_evict options to false and setting client_reconnect_stale to true on the client side, the file system can be accessed even if eviction occurs. It seemed like there was no major problem accessing the file system as the session was still attached. What I'm curious about is if I turn on the above option, will there be any other side effects? For example, should I take some action if the integrity of the file is broken or if there is an issue on the mds side? I am asking a question because there are no details regarding this in the official CephFS documentation. Thank you Yongseok

2 2