- ceph-users - lists.ceph.io

Call for Interest: Managed SMB Protocol Support
by John Mulligan 11 Apr '24

11 Apr '24

Hello Ceph List, I'd like to formally let the wider community know of some work I've been involved with for a while now: adding Managed SMB Protocol Support to Ceph. SMB being the well known network file protocol native to Windows systems and supported by MacOS (and Linux). The other key word "managed" meaning integrating with Ceph management tooling - in this particular case cephadm for orchestration and eventually a new MGR module for managing SMB shares. The effort is still in it's very early stages. We have a PR adding initial support for Samba Containers to cephadm [1] and a prototype for an smb MGR module [2]. We plan on using container images based on the samba-container project [3] - a team I am already part of. What we're aiming for is a feature set similar to the current NFS integration in Ceph, but with a focus on bridging non-Linux/Unix clients to CephFS using a protocol built into those systems. A few major features we have planned include: * Standalone servers (internally defined users/groups) * Active Directory Domain Member Servers * Clustered Samba support * Exporting Samba stats via Prometheus metrics * A `ceph` cli workflow loosely based on the nfs mgr module I wanted to share this information in case there's wider community interest in this effort. I'm happy to take your questions / thoughts / suggestions in this email thread, via Ceph slack (or IRC), or feel free to attend a Ceph Orchestration weekly meeting! I try regularly attend and we sometimes discuss design aspects of the smb effort there. It's on the Ceph Community Calendar. Thanks! [1] - https://github.com/ceph/ceph/pull/55068 [2] - https://github.com/ceph/ceph/pull/56350 [3] - https://github.com/samba-in-kubernetes/samba-container/ Thanks for reading, --John Mulligan

7 16

Cephadm host keeps trying to set osd_memory_target to less than minimum
by mads2a＠gmail.com 11 Apr '24

11 Apr '24

I have a virtual ceph cluster running 17.2.6 with 4 ubuntu 22.04 hosts in it, each with 4 OSD's attached. The first 2 servers hosting mgr's have 32GB of RAM each, and the remaining have 24gb For some reason i am unable to identify, the first host in the cluster appears to constantly be trying to set the osd_memory_target variable to roughly half of what the calculated minimum is for the cluster, i see the following spamming the logs constantly Unable to set osd_memory_target on my-ceph01 to 480485376: error parsing value: Value '480485376' is below minimum 939524096 Default is set to 4294967296. I did double check and osd_memory_base (805306368) + osd_memory_cache_min (134217728) adds up to minimum exactly osd_memory_target_autotune is currently enabled. But i cannot for the life of me figure out how it is arriving at 480485376 as a value for that particular host that even has the most RAM. Neither the cluster or the host is even approaching max utilization on memory, so it's not like there are processes competing for resources.

3 11

Have a problem with haproxy/keepalived/ganesha/docker
by ruslan.nurabayev＠kcell.kz 11 Apr '24

11 Apr '24

Hello! I've install my 5-node CEPH cluster next install NFS server by command: ceph nfs cluster create nfshacluster 5 --ingress --virtual_ip 192.168.171.48/26 --ingress-mode haproxy-protocol. I don't understand fully how this must be works but when i stop NFS daemon even on one of this nodes i've see that writing on NFS shares is disappear (testing via vdbench). As i understand it is wrong and IO from stopped daemon must swithcing to another NFS daemon without any impact on IO. Can someone help me with troubleshoot this issue? Or explain how done full-fledged Active-Active HA NFS Cluster for production use.

1 0

RGW/Lua script does not show logs
by soyoon.lee＠samsung.com 11 Apr '24

11 Apr '24

Hello, I wrote a Lua script in order to retrieve RGW logs such as bucket name, bucket owner, etc. However, when I apply a lua script I wrote using the below command, I do not see any logs start with Lua: INFO radosgw-admin script put --infile=/usr/tmp/testPreRequest.lua --context=postrequest <my lua script> function print_bucket_log(bucket) RGWDebugLog(" Name: " .. bucket.Name) end if Request.Bucket then RGWDebugLog("bucket operation logs: ") print_bucket_log(Request.Bucket) end According to the official document regarding Lua Scripting, The RGWDebugLog() function accepts a string and prints it to the debug log with priority 20. Each log message is prefixed Lua INFO:. This function has no return value. even though I set debug_rgw = 20, I do not see any logs. However, if when I apply the below lua script with bucket.Id, I am getting Lua: ERROR like below: Lua ERROR: [string "function print_bucket_log(bucket)..."]:3: attempt to concatenate a nil value (field 'Id') <my lua script> function print_bucket_log(bucket) RGWDebugLog(" Name: " .. bucket.Name) RGWDebugLog(" Id: " .. bucket.Id) end if Request.Bucket then RGWDebugLog("bucket operation logs: ") print_bucket_log(Request.Bucket) end Any help would be very appreciated!

3 2

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

Strange placement groups warnings
by Dmitriy Maximov 10 Apr '24

10 Apr '24

Dear Ceph experts, recently we have upgraded our ceph cluster from octopus (15.2.17) to pacific (16.2.14 and then to 16.2.15). Just after upgrade warnings that all (except device_health_metrics pool) our pools have too many placement groups appeared. This warning looks like generated by autoscaler, but there was no suggestions from it. When I started to investigate situation, I turned bulk flag to true for pools intended to store large amount of data (just to look changes). After that most pools reports that they have too few placement groups, but there is still no suggestions from autoscaler (NEW PG_NUM column is empty) # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK metadata 22567M 3.0 57255G 0.0012 1.0 32 warn False rbd_hdd02_ec_meta 20817k 3.0 365.3T 0.0000 1.0 32 warn False device_health_metrics 1002M 3.0 205.4T 0.0000 1.0 1 on False rbd_ssd_ec_meta 0 3.0 57255G 0.0000 1.0 32 warn False cmd_spool 45094G 1.3333333730697632 205.4T 0.2859 1.0 512 warn True data 60496G 3.0 365.3T 0.4851 1.0 512 warn True rbd 2625G 3.0 205.4T 0.0374 1.0 64 warn True kedr_spool 6900G 10240G 1.3333333730697632 205.4T 0.0649 1.0 128 warn True gcf_spool 1119M 1.3333333730697632 205.4T 0.0000 1.0 32 warn True rbd_ssd 107.8G 3.0 57255G 0.0056 1.0 32 warn True data_ssd 775.8G 3.0 57255G 0.0407 1.0 128 warn True only warnings in 'ceph status' and 'ceph health detail' # ceph health detail HEALTH_WARN 6 pools have too few placement groups; 4 pools have too many placement groups [WRN] POOL_TOO_FEW_PGS: 6 pools have too few placement groups Pool data has 512 placement groups, should have 2048 Pool rbd has 64 placement groups, should have 512 Pool kedr_spool has 128 placement groups, should have 256 Pool gcf_spool has 32 placement groups, should have 256 Pool rbd_ssd has 32 placement groups, should have 2048 Pool data_ssd has 128 placement groups, should have 2048 [WRN] POOL_TOO_MANY_PGS: 4 pools have too many placement groups Pool metadata has 32 placement groups, should have 32 Pool rbd_hdd02_ec_meta has 32 placement groups, should have 32 Pool rbd_ssd_ec_meta has 32 placement groups, should have 32 Pool cmd_spool has 512 placement groups, should have 512 When I changed number of placement groups of pool kedr_spool from 128 to 256 as suggested in warning, data rebalanced successfully, but now pool warns that it have too many placement groups: Pool kedr_spool has 256 placement groups, should have 256. The most strange for me in these warnings is that the current number of PGs is the same that should be. What is it and what how to solve these warnings? Best regards, Dmitriy

1 0

Issue about execute "ceph fs new"
by elite_stu＠163.com 10 Apr '24

10 Apr '24

Everything goes fine except execute "ceph fs new kingcephfs cephfs-king-metadata cephfs-king-data", its shows 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds. But i see there is one mds services running, please help me to fix the issue, thanks a lot. bash-4.4$ bash-4.4$ ceph fs new kingcephfs cephfs-king-metadata cephfs-king-data new fs with metadata pool 7 and data pool 8 bash-4.4$ bash-4.4$ ceph -s cluster: id: de9af3fe-d3b1-4a4b-bf61-929a990295f6 health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 3 daemons, quorum a,b,c (age 2d) mgr: a(active, since 2d), standbys: b mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 2d), 3 in (since 2d) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 2/2 healthy pools: 14 pools, 233 pgs objects: 592 objects, 450 MiB usage: 1.5 GiB used, 208 GiB / 210 GiB avail pgs: 233 active+clean io: client: 921 B/s rd, 1 op/s rd, 0 op/s wr bash-4.4$ bash-4.4$ ceph fs status myfs - 0 clients ==== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active myfs-a Reqs: 0 /s 10 13 12 0 0-s standby-replay myfs-b Evts: 0 /s 0 3 2 0 POOL TYPE USED AVAIL myfs-metadata metadata 180k 65.9G myfs-replicated data 0 65.9G kingcephfs - 0 clients ========== POOL TYPE USED AVAIL cephfs-king-metadata metadata 0 65.9G cephfs-king-data data 0 65.9G MDS version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) bash-4.4$ bash-4.4$ bash-4.4$ bash-4.4$ bash-4.4$ ceph mds stat myfs:1 kingcephfs:0 {myfs:0=myfs-a=up:active} 1 up:standby-replay bash-4.4$ bash-4.4$ ceph fs ls name: myfs, metadata pool: myfs-metadata, data pools: [myfs-replicated ] name: kingcephfs, metadata pool: cephfs-king-metadata, data pools: [cephfs-king-data ] bash-4.4$

2 6

Regarding write on CephFS - Operation not permitted
by elite_stu＠163.com 10 Apr '24

10 Apr '24

Dears , CephFS mount succesful, however i encounted an issue about write file with error message "Operation not permitted". It happend even the file chmod 777, please help me to solve, thanks a lot! [root@vm-04 mycephfs]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/centos-root xfs 46G 1.3G 44G 3% / devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs tmpfs 1.9G 8.5M 1.9G 1% /run tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/vda1 xfs 1014M 145M 870M 15% /boot tmpfs tmpfs 379M 0 379M 0% /run/user/0 192.168.100.7:6789,192.168.100.8:6789,192.168.100.9:6789:/ ceph 19G 0 19G 0% /mnt/mycephfs [root@vm-04 mycephfs]# [root@vm-04 mycephfs]# pwd /mnt/mycephfs [root@vm-04 mycephfs]# [root@vm-04 mycephfs]# ls -rlt total 0 -rw-r--r-- 1 root root 0 Apr 8 20:44 aa.txt -rw-r--r-- 1 root root 0 Apr 8 20:54 bb.txt [root@vm-04 mycephfs]# cp aa.txt cc.txt cp: error reading ‘aa.txt’: Operation not permitted cp: failed to extend ‘cc.txt’: Operation not permitted [root@vm-04 mycephfs]# [root@vm-04 mycephfs]# echo "123" > aa.txt -bash: echo: write error: Operation not permitted [root@vm-04 mycephfs]# [root@vm-04 mycephfs]#

1 1

Ceph alert module different code path?
by Torkil Svensgaard 09 Apr '24

09 Apr '24

Hi We are getting a lot of odd warnings from the alert module by email, alerts which never seem to show up on "ceph -s". Here's a somewhat funny one: Date: Tue, 09 Apr 2024 16:48:06 -0000 " --- New --- [WARN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering pg 11.a0c is stuck peering since forever, current state peering, last acting [504,491,264,535,157,297] " Date: Tue, 09 Apr 2024 16:48:11 -0000 " --- Cleared --- [WARN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering pg 11.a0c is stuck peering since forever, current state peering, last acting [504,491,264,535,157,297] " They often come in pairs like this, first a new (ghost?) warning and then a cleared warning right after. Since these never show up on "ceph -s" I am guessing this a separate code path and the alert module is somewhat broken? Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

1 0

slow pwrite64()s to ceph
by Kelly, Mark (RIS-BCT) 09 Apr '24

09 Apr '24

Hi, Thank you for any time. We are tracking some very slow pwrite64() calls to a ceph filesystem - 20965 11:04:24.049186 <... pwrite64 resumed>) = 65536 <4.489594> 20966 11:04:24.069765 <... pwrite64 resumed>) = 65536 <4.508859> 20967 11:04:24.090354 <... pwrite64 resumed>) = 65536 <4.510256> But other pwrite64()s from the same program in other threads to other files on the same ceph fs seem fine; we cannot really reproduce this, but it happens occasionally. It seems we are spending some time in ceph_aio_write() when this is happening (see call graph below) I've noticed THP (transparent Huge Pages) is enabled. We are running version 15.2.17 on CentOS 7.9 Do not seem to be under any significant memory pressure when this happens, just many threads of this app blocked on i/o in pwrite64()s. I am suggesting an upgrade, but until then, do you think this situation involves ceph and could be improved if we disable THP ? Thanks for any advice or suggestion, -mark Call graph of app when slow pwrites64()s are happening - --87.08%--system_call_fastpath |--58.20%--sys_pwrite64 | --58.03%--vfs_write | do_sync_write | ceph_aio_write | |--54.33%--generic_file_buffered_write | | |--27.09%--ceph_write_begin | | | |--17.01%--grab_cache_page_write_begin | | | | |--7.51%--add_to_page_cache_lru | | | | | |--4.28%--__add_to_page_cache_locked | | | | | | --3.95%--mem_cgroup_cache_charge | | | | | | mem_cgroup_charge_common | | | | | | --3.75%--__mem_cgroup_commit_charge | | | | | --3.23%--lru_cache_add | | | | | __lru_cache_add | | | | | --2.94%--pagevec_lru_move_fn | | | | | |--0.70%--mem_cgroup_page_lruvec | | | | | |--0.67%--__pagevec_lru_add_fn | | | | | --0.57%--release_pages | | | | |--5.31%--__page_cache_alloc | | | | | --5.00%--alloc_pages_current | | | | | __alloc_pages_nodemask | | | | | --4.23%--get_page_from_freelist | | | | | |--1.85%--__rmqueue | | | | | | --1.49%--list_del | | | | | | __list_del_entry | | | | | --1.74%--list_del | | | | | __list_del_entry | | | | --3.92%--__find_lock_page | | | | __find_get_page | | | | --3.46%--radix_tree_lookup_slot | | | | |--2.76%--radix_tree_descend | | | | --0.70%--__radix_tree_lookup | | | | radix_tree_descend | | | --9.45%--ceph_update_writeable_page | | | --8.94%--readpage_nounlock | | | --8.60%--ceph_osdc_readpages | | | --8.18%--submit_request | | | __submit_request | | | calc_target.isra.50 | | | ceph_pg_to_up_acting_osds | | | crush_do_rule | | | crush_choose_firstn | | | |--4.89%--crush_choose_firstn | | | | is_out.isra.2.part.3 | | | --3.30%--crush_bucket_choose | | |--14.50%--copy_user_enhanced_fast_string | | |--6.34%--ceph_write_end | | | |--3.76%--set_page_dirty | | | | ceph_set_page_dirty | | | | --2.86%--__set_page_dirty_nobuffers | | | | |--0.92%--_raw_spin_unlock_irqrestore | | | | --0.59%--radix_tree_tag_set | | | --1.88%--unlock_page | | | __wake_up_bit | | |--3.44%--iov_iter_fault_in_readable | | --2.70%--mark_page_accessed | |--2.54%--mutex_lock | | __mutex_lock_slowpath | | --2.14%--schedule_preempt_disabled | | __schedule | | | | | |--0.82%--finish_task_switch | | | __perf_event_task_sched_in | | | perf_pmu_enable | | | x86_pmu_enable | | | --0.80%--intel_pmu_enable_all | | | --0.74%--__intel_pmu_enable_all.isra.23 | | | --0.69%--native_write_msr_safe | | --0.75%--__perf_event_task_sched_out | --0.56%--mutex_unlock | __mutex_unlock_slowpath ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

1 0