November 2023 - ceph-users

[CEPH] OSD Memory Usage
by Nguyễn Hữu Khôi 16 Nov '23

16 Nov '23

Hello, I am using a CEPH cluster. After monitoring it, I set: ceph config set osd osd_memory_target_autotune false ceph config set osd osd_memory_target 1G Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case? Ceph version: Quincy OSD: 3 nodes with 11 nvme each and 512GB ram per node. CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket. Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000 Thank you very much. Nguyen Huu Khoi

3 8

remove spurious data
by Giuliano Maggi 16 Nov '23

16 Nov '23

Hi, I’d like to remove some “spurious" data: root@nerffs03:/# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.0 PiB 1.0 PiB 47 GiB 47 GiB 0 TOTAL 1.0 PiB 1.0 PiB 47 GiB 47 GiB 0 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 70 MiB 19 209 MiB 0 327 TiB .nfs 3 32 572 B 4 36 KiB 0 327 TiB root@nerffs03:/# as you can see, there are no data pools. The 47GiB could be from previous pools/filesystems that I used for testing. how can I remove this “spurious” data without affecting the OSDs? I was checking "ceph-volume lvm zap”, but I am not sure it this is the right one. (I am running ceph version quincy 17.2.7) Thanks, Giuliano.

2 1

planning upgrade from pacific to quincy
by Simon Oosthoek 16 Nov '23

16 Nov '23

Hi All (apologies if you get again, I suspect mails from my @science.ru.nl account get dropped by most receiving mail servers, due to the strict DMARC policy (p=reject) in place) after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we are now planning to upgrade our cluster which is running Pacific (at least on the mons/mdss/osds, the gateways are by accident running quincy already). The installation is via packages from ceph.com, unless it's quincy from ubuntu. ceph versions: "mon": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "mgr": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "osd": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 252, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12 }, "mds": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 2 }, "rgw": {"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 }, "overall": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 260, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 } The OS on the mons and mdss are still ubuntu 18.04, the osds are a mix of ubuntu 18 and ubuntu 20. The gateways are ubuntu 22.04, which is why these are already on quincy. The plan is to move to quincy and eventually cephadm/containered ceph, since that is apparently "the way to go", though I have my doubts. The steps we think are the right order are: - reinstall the mons with ubuntu 22.04 + quincy - reinstall the osds (same) - reinstall the mdss (same) Once this is up and running, we want to investigate and migrate to cephadm orchestration. Alternative appear to be: move to orchestration first and then upgrade ceph to quincy (possibly skipping the ubuntu upgrade?) Another alternative could be to upgrade to quincy on ubuntu 18.04 using packages, but I haven't investigated the availability of quincy packages for ubuntu 18.04 (which is out of free (LTS) support by canonical) Cheers /Simon -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

1 0

Ceph Leadership Team Meeting Minutes Nov 15, 2023
by Ernesto Puerta 16 Nov '23

16 Nov '23

Hi Cephers, These are the topics discussed today: - 18.2.1 - Almost ready, packages built/signed - Plan to release on Monday - Last minute PR for Rook - Lab update to be finished by tomorrow - Finalize CDM APAC time - Review https://doodle.com/meeting/participate/id/aM9XGZ3a/vote?reauth=true - New time: 9.30 Pacific Time - Laura and Neha will sync up about changing the time - Squid dev freeze date - Proposal: end of January - Docs question: https://tracker.ceph.com/issues/11385: Can a member of the community just raise a PR attempting to standardize commands, without coordinating with a team? - ballpark date for pacific eol? docs still have the '2023-10-01' estimate - Discussion regarding EOL being at time of release or at some point shortly after to account for regressions - Conversation regarding clarity of messaging being important re: expectations for post-release fixes - concerns on stability of minor releases - tentatively this year - distro status update: still working to remove centos8/rhel8/ubuntu20 from main. https://github.com/ceph/ceph/pull/53901 stalled on container stuff in teuthology, and the need to rebuild containers with centos9 base - discuss the quincy/dashboard-v3 backports? was tabled from 11/1 [postponed to Nov 22] - Docs (Zac): CQ January 2024 https://pad.ceph.com/p/ceph_quarterly_2024_01 - Unittestability of dencoding: ask for review -- https://trello.com/c/R0h47dq2/870-unittestability-of-dencoding#comment-6553… - User + Dev meeting tomorrow - Need RGW representatives and people with EC profile knowledge Kind Regards, Ernesto

2 1

iSCSI GW trusted IPs
by Ramon Orrù 16 Nov '23

16 Nov '23

Hi, I’m configuring the iSCSI GW services on a quincy 17.2.3 cluster. I brought almost everything up and running (using cephadm), but I’m stuck in a configuration detail: if I check the gateway status in the Block -> iSCSI -> Overview section of the dashboard, they’re showing “Down” status, while the gateways are actually running. It makes me think the mgr is not able to talk with iSCSI APIs in order to collect info on the gateways, despite I correctly added my mgr hosts IPs to the trusted_ip_list parameter in my iscsi service definition yaml. While further checking the gateway logs I found some messages like: debug ::ffff:172.17.17.22 - - [15/Nov/2023 10:54:05] "GET /api/config?decrypt_passwords=True HTTP/1.1" 200 - debug ::ffff:172.17.17.22 - - [15/Nov/2023 10:54:05] "GET /api/_ping HTTP/1.1" 200 - debug ::ffff:172.17.17.22 - - [15/Nov/2023 10:54:05] "GET /api/gatewayinfo HTTP/1.1" 200 - Just after I reload the dashboard page. So I tried to add the 172.17.17.22 IP address to trusted_ip_list and it worked: iSCSI gateways status went green and Up on the dashboard. It sounds to me like it's some container private network address, but I can’t find any evidence of it when inspecting the containers cephadm spawned. My question is: how can I identify the IPs I need to make the iSCSI gateways properly reachable? I tried to add the whole 172.16.0.0/24 private class but no luck , the iscsi container starts but is not allowing 172.17.17.22 to access the APIs. Thanks in advance regards Ramon

3 2

migrate wal/db to block device
by Chris Dunlop 16 Nov '23

16 Nov '23

Hi, What's the correct way to migrate an OSD wal/db from a fast device to the (slow) block device? I have an osd with wal/db on a fast LV device and block on a slow LV device. I want to move the wal/db onto the block device so I can reconfigure the fast device before moving the wal/db back to the fast device. This link says to use "ceph-volume lvm migrate" (I'm on pacific, but the quincy and reef docs are the same): https://docs.ceph.com/en/pacific/ceph-volume/lvm/migrate/ I tried: $ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-volume lvm migrate --osd-id ${osdid} --osd-fsid ${osd_fsid} \ --from db wal --target ${block_vglv} $ systemctl stop ${osd_service} $ systemctl start ${osd_service} "cephadm ceph-volume lvm list" now shows only the (slow) block device whereas before the migrate it was showing both the block and db devices. However "lsof" shows the new osd process still has the original fast wal/db device open and "iostat" shows this device is still getting i/o. Also: $ ls -l /var/lib/ceph/${fsid}/osd.${osdid}/block* ...shows both the "block" and "block.db" symlinks to the original separate devices. And there are now no lv_tags on the original wal/db LV: $ lvs -o lv_tags ${original_db_vg_lv} Now I'm concerned there's device mismatch for this osd: "cephadm ceph-volume lvm list" believes there's no separate wal/db, but the osd is currently *using* the original separate wal/db. I guess if the server were to restart this osd would be in all sorts of trouble. What's going on there, and what can be done to fix it? Is it a matter of recreating the tags on the original db device? (But then what happens to whatever did get migrated to the block device - e.g. is that space lost?) Or is it a matter of using ceph-bluestore-tool to do a bluefs-bdev-migrate, e.g. something like: $ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ osddir=/var/lib/ceph/osd/ceph-${osdid} $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-bluestore-tool --path ${osddir} --devs-source ${osddir}/block.db \ --dev-target ${osddir}/block bluefs-bdev-migrate $ rm /var/lib/ceph/${fsid}/osd.${osdid}/block.db $ systemctl stop ${osd_service} $ systemctl start ${osd_service} Or... something else? And how *should* moving the wal/db be done? Cheers, Chris

3 6

Large size differences between pgs
by Miroslav Svoboda 15 Nov '23

15 Nov '23

Hi,is it possible decrease large size differences between pgs? I have 5PB cluster and differences between smalest and bigest pgs are somewhere about 25GB.thanks,Svoboda Miroslav

1 0

per-rbd snapshot limitation
by Wesley Dillingham 15 Nov '23

15 Nov '23

looking into how to limit snapshots at the ceph level for RBD snapshots. Ideally ceph would enforce an arbitrary number of snapshots allowable per rbd. Reading the man page for rbd command I see this option: https://docs.ceph.com/en/quincy/man/8/rbd/#cmdoption-rbd-limit --limit Specifies the limit for the number of snapshots permitted. Seems perfect. But on attempting to use it as such I get an error: admin@rbdtest:~$ rbd create testpool/test3 --size=100M --limit=3 rbd: unrecognised option '--limit=3' Where am I going wrong here? Is there another way to enforce a limit of snapshots for RBD? Thanks. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

3 5

Join us for the User + Dev Monthly Meetup - November 16!
by Laura Flores 15 Nov '23

15 Nov '23

Hi Ceph users and developers, You are invited to join us at the User + Dev meeting this week Thursday, November 16th at 10:00 AM EST! See below for more meeting details. The focus topic, "Operational Reliability and Flexibility in Ceph Upgrades", will be presented by Christian Theune. His presentation will highlight some issues encountered in a long running cluster when upgrading to stable releases, including migration between EC profiles, challenges related to RGW zone replication, and low-level bugs that need more attention from developers. The last part of the meeting will be dedicated to open discussion. Feel free to add questions for the speakers or additional topics under the "Open Discussion" section on the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have an idea for a focus topic you'd like to present at a future meeting, you are welcome to submit it to this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Any Ceph user or developer is eligible to submit! Thanks, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time conversions: UTC: Thursday, November 16, 15:00 UTC Mountain View, CA, US: Thursday, November 16, 7:00 PST Phoenix, AZ, US: Thursday, November 16, 8:00 MST Denver, CO, US: Thursday, November 16, 8:00 MST Huntsville, AL, US: Thursday, November 16, 9:00 CST Raleigh, NC, US: Thursday, November 16, 10:00 EST London, England: Thursday, November 16, 15:00 GMT Paris, France: Thursday, November 16, 16:00 CET Helsinki, Finland: Thursday, November 16, 17:00 EET Tel Aviv, Israel: Thursday, November 16, 17:00 IST Pune, India: Thursday, November 16, 20:30 IST Brisbane, Australia: Friday, November 17, 1:00 AEST Singapore, Asia: Thursday, November 16, 23:00 +08 Auckland, New Zealand: Friday, November 17, 4:00 NZDT -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

rasize= in ceph.conf some section?
by Pat Riehecky 15 Nov '23

15 Nov '23

Hello, I'm trying to make it easy to distribute the expected config settings for my ceph volumes for other admin groups. Is there a place I can set rasize in the ceph.conf where the client would pick it up? The NFS world these days has nfsmount.conf that I've grown very fond of. I'm a bit concerned that there will be drift in rasize and whatnot that causes performance issues. While it doesn't seem much, having two places to look for mount information is less ideal....

1 0

ceph -s very slow in my rdma eviroment
by WeiGuo Ren 15 Nov '23

15 Nov '23

Today I run some ceph -s in my rdma enviroment. but very slow. After perf ceph -s to FlameGraph. I find almost all the time is spent on compact zone. Has anyone encountered it? my enviroment . rpm -qa | grep ibverb libibverbs-41mlnx1-OFED.4.1.0.1.1.41102.x86_64 libibverbs-devel-41mlnx1-OFED.4.1.0.1.1.41102.x86_64 ceph -v 14.2.5

1 0

planning upgrade from pacific to quincy
by Simon Oosthoek 15 Nov '23

15 Nov '23

Hi All (apologies if you get this twice, I suspect mails from my @science.ru.nl account get dropped by most receiving mail servers, due to the strict DMARC policies in place) after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we are now planning to upgrade our cluster which is running Pacific (at least on the mons/mdss/osds, the gateways are by accident running quincy already). The installation is via packages from ceph.com, unless it's quincy from ubuntu. ceph versions: "mon": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "mgr": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "osd": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 252, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12 }, "mds": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 2 }, "rgw": {"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 }, "overall": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 260, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 } The OS on the mons and mdss are still ubuntu 18.04, the osds are a mix of ubuntu 18 and ubuntu 20. The gateways are ubuntu 22.04, which is why these are already on quincy. The plan is to move to quincy and eventually cephadm/containered ceph, since that is apparently "the way to go", though I have my doubts. The steps we think are the right order are: - reinstall the mons with ubuntu 22.04 + quincy - reinstall the osds (same) - reinstall the mdss (same) Once this is up and running, we want to investigate and migrate to cephadm orchestration. Alternative appear to be: move to orchestration first and then upgrade ceph to quincy (possibly skipping the ubuntu upgrade?) Another alternative could be to upgrade to quincy on ubuntu 18.04 using packages, but I haven't investigated the availability of quincy packages for ubuntu 18.04 (which is out of free (LTS) support by canonical) Cheers /Simon

1 0

planning upgrade from pacific to quincy
by Simon Oosthoek 15 Nov '23

15 Nov '23

Hi All after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we are now planning to upgrade our cluster which is running Pacific (at least on the mons/mdss/osds, the gateways are by accident running quincy already). The installation is via packages from ceph.com, unless it's quincy from ubuntu. ceph versions: "mon": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "mgr": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "osd": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 252, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12 }, "mds": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 2 }, "rgw": {"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 }, "overall": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 260, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 } The OS on the mons and mdss are still ubuntu 18.04, the osds are a mix of ubuntu 18 and ubuntu 20. The gateways are ubuntu 22.04, which is why these are already on quincy. The plan is to move to quincy and eventually cephadm/containered ceph, since that is apparently "the way to go", though I have my doubts. The steps we think are the right order are: - reinstall the mons with ubuntu 22.04 + quincy - reinstall the osds (same) - reinstall the mdss (same) Once this is up and running, we want to investigate and migrate to cephadm orchestration. Alternative appear to be: move to orchestration first and then upgrade ceph to quincy (possibly skipping the ubuntu upgrade?) Another alternative could be to upgrade to quincy on ubuntu 18.04 using packages, but I haven't investigated the availability of quincy packages for ubuntu 18.04 (which is out of free (LTS) support by canonical) Cheers /Simon

1 0

RGW: user modify default_storage_class does not work
by Huy Nguyen 15 Nov '23

15 Nov '23

Hi community, I'm using Ceph version 16.2.13. I tried to set default_storage_class but seems like it didn't work. Here is steps I did: I already had a storage class name COLD, then I modify the user default_storage_class like this: radosgw-admin user modify --uid testuser --placement-id default-placement --storage-class COLD after that, user info has show correctly: radosgw-admin user info --uid testuser { ... "op_mask": "read, write, delete", "default_placement": "default-placement", "default_storage_class": "COLD", ... Then I put a file using boto3, without specify any storage class: s3.Object(bucket_name, 'testdefault-object').put(Body="0"*1000)) But the object still jump into the STANDARD storage class. I don't know if this is a bug or did I miss something? Thanks

2 2

Stretch mode size
by Eugen Block 15 Nov '23

15 Nov '23

Hi, I'd like to ask for confirmation how I understand the docs on stretch mode [1]. It requires exact size 4 for the rule? Other sizes are not supported/won't work, for example size 6? Are there clusters out there which use this stretch mode? Once stretch mode is enabled, it's not possible to get out of it. How would one deal with a burnt down datacenter which can take months to rebuild? In a "self-managed" stretch cluster (let's say size 6) I could simply change the crush rule to not consider the failed datacenter anymore, deploy an additional mon somewhere and maybe reduce the size/min_size. Am I missing something? Thanks, Eugen [1] https://docs.ceph.com/en/reef/rados/operations/stretch-mode/#id2

2 4

Ceph Allocation - used space is unreasonably higher than stored space
by Motahare S 14 Nov '23

14 Nov '23

Hello everyone, Recently we have noticed that the results of "ceph df" stored and used space does not match; as the amount of stored data *1.5 (ec factor) is still like 5TB away from used amount: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL default.rgw.buckets.data 12 1024 144 TiB 70.60M 221 TiB 18.68 643 TiB blob and alloc configs are as below: bluestore_min_alloc_size_hdd : 65536 bluestore_min_alloc_size_ssd : 4096 luestore_max_blob_size_hdd : 524288 bluestore_max_blob_size_ssd : 65536 bluefs_shared_alloc_size : 65536 From sources across web about how ceph actually writes on the disk, I presumed that It will zero-pad the extents of an object to match the 4KB bdev_block_size, and then writes it in a blob which matches the min_alloc_size, however it can re-use parts of the blob's unwritten (but allocated because of min_alloc_size) space for another extent later. The problem though, was that we tested different configs in a minimal ceph octopus cluster with a 2G osd and bluestore_min_alloc_size_hdd = 65536. When we uploaded a 1KB file with aws s3 client, the amount of used/stored space was 64KB/1KB. We then uploaded another 1KB, and it went 128K/2K; kept doing it until 100% of the pool was used, but only 32MB stored. I expected ceph to start writing new 1KB files in the wasted 63KB(60KB)s of min_alloc_size blocks, but the cluster was totally acting as a full cluster and could no longer receive any new object. Is this behaviour expected for s3? Does ceph really use 64x space if your dataset is made of 1KB files? and all your object sizes should be a multiple of 64KB? Note that 5TB / (70.6M*1.5) ~ 50 so for every rados object about 50KB is wasted on average. we didn't observe this problem in RBD pools, probably because it cuts all objects in 4MB. I know that min_alloc_hdd is changed to 4KB in pacific, but I'm still curious how allocation really works and why it doesn't behave as expected? Also, re-deploying OSDs is a headache. Sincerely Motahare

3 3

CephFS mirror very slow (maybe for small files?)
by Stuart Cornell 14 Nov '23

14 Nov '23

Hi all. I have successfully configured an operational mirror between 2 sites for Ceph FS. The mirroring is running but the speed of data transfer is varying a lot over time (200KB/s – 120MB/s). The network infrastructure between the two Ceph clusters is reliable and should not be the cause of this speed variation. The FS in question, has a lot of small files in it and I suspect this is the cause of the variability – ie, the transfer of many small files will be more impacted by greater site-site latency. If this suspicion is true, what options do I have to improve the overall throughput? * Is it possible to parallelise or “chunk” the transfers with some options to the mirror daemon? * Would the use of multiple snapshot mirror points help? * Note that I am currently forced to use only a single point ( ceph fs snapshot mirror add <fs_name> <path> ) because the part of the FS in use is managed by Openstack Manila which created a subvolume. Requests to add mirrors for sub-directories are therefore denied. Any suggestions for how I can improve this throughput would be most welcome. Currrently I am running Pacific (16.2.10) on the sender and Quincy (17.2.6) on the target. Stuart Cornell Cloud Development Director http://graphcore.ai ** We have updated our privacy policy, which contains important information about how we collect and process your personal data. To read the policy, please click here<http://www.graphcore.ai/privacy> ** This email and its attachments are intended solely for the addressed recipients and may contain confidential or legally privileged information. If you are not the intended recipient you must not copy, distribute or disseminate this email in any way; to do so may be unlawful. Any personal data/special category personal data herein are processed in accordance with UK data protection legislation. All associated feasible security measures are in place. Further details are available from the Privacy Notice on the website and/or from the Company. Graphcore Limited (registered in England and Wales with registration number 10185006) is registered at 107 Cheapside, London, UK, EC2V 6DN. This message was scanned for viruses upon transmission. However Graphcore accepts no liability for any such transmission.

3 4

reduce mds_beacon_interval and mds_beacon_grace
by Dmitry Melekhov 14 Nov '23

14 Nov '23

Hello! I guess that default 4 and 15 seconds are too long for my configuration. If I want to reduce mds_beacon_grace to 5 seconds is 1 right value for mds_beacon_interval ? Thank you!

1 0

shrink db size
by Curt 13 Nov '23

13 Nov '23

Hello, As far as I can tell there is no way to shrink a db/wal after creation. I recently added a new server to my cluster with SSD's for the wal/db and just used the ceph dashboard for deployment. I did not specify a db size, which is my mistake, it seems by default it uses "block.db has no size configuration, will fallback to using as much as possible". So now my issue is I added 2 more drives, but with no space left on the SSD's get "2 fast devices were passed, but none are available". I had 7 4TB HDD and 2 2Tb SSD's and added 2 more 4TB. Distribution is 4 and 3 between the 2 SSD's. I just want to confirm, the best option is to set the bluestore_block_db_size/wal_size, zap and let them be recreated with a size of 300 and 2. I chose those just because I have space. I'm not going to do them all at the same time. Cheers, Curt

1 0

CEPH Cluster mon is out of quorum
by Mosharaf Hossain 13 Nov '23

13 Nov '23

Dear Concern I am observing a mon is out of quorum. The current running version of Ceph is octopus. Total Node in the cluster: 13 Mon: 3/3 Network: 20G(10Gx 2) bonded link Each node capacity: 512GB RAM + 72 Core CPU root@ceph6:/var/run/ceph# systemctl status ceph-mon(a)ceph6.service â— ceph-mon(a)ceph6.service - Ceph cluster monitor daemon Loaded: loaded (/lib/systemd/system/ceph-mon@.service; indirect; vendor preset: enabled) Active: active (running) since Mon 2023-11-13 14:18:28 +06; 55min ago Main PID: 165299 (ceph-mon) Tasks: 26 CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon(a)ceph6.service â””â”€165299 /usr/bin/ceph-mon -f --cluster ceph --id ceph6 --setuser ceph --setgroup ceph Nov 13 15:13:09 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:09.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:14 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:14.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:34 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:34.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:39 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:39.191+0600 7 root@ceph6:/var/run/ceph# ceph -s cluster: id: f8096ec7-51db-4557-85e6-57d7fdfe9423 health: HEALTH_WARN 1/3 mons down, quorum ceph2,mon1 nodeep-scrub flag(s) set 614 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph2,mon1 (age 8h), out of quorum: ceph6 mgr: ceph4(active, since 2w), standbys: mon1, ceph3, ceph1, ceph6 mds: cephfs:1 {0=ceph8=up:active} 1 up:standby osd: 107 osds: 102 up (since 3h), 102 in (since 3h); 24 remapped pgs flags nodeep-scrub rgw: 4 daemons active (ceph10.rgw0, ceph7.rgw0, ceph9.rgw0, mon1.rgw0) rgw-nfs: 2 daemons active (ceph7, ceph9) root@ceph6:/var/run/ceph# ceph daemon mon.ceph6 mon_status { "name": "ceph6", "rank": 2, "state": "probing", "election_epoch": 0, "quorum": [], "features": { "required_con": "2449958747315978244", "required_mon": [ "kraken", "luminous", Total 3 mon in the cluster and all addresses are able to ping and telnet between them. 1. mon1: 10.10.10.71 2. ceph2: 10.10.10.52 3. ceph6: 10.10.10.56 Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain(a)bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts…>

2 1

OSD disk is active in node but ceph show osd down and out
by Mosharaf Hossain 13 Nov '23

13 Nov '23

Hello In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, despite all SSD disks appearing to be active on the node. What might be the cause of this issue? #ceph osd tree [showing disk for ceph6 status] -11 19.20854 host ceph6 10 ssd 1.74609 osd.10 down 0 1.00000 35 ssd 1.74609 osd.35 up 1.00000 1.00000 40 ssd 1.74609 osd.40 down 0 1.00000 45 ssd 1.74609 osd.45 down 0 1.00000 50 ssd 1.74609 osd.50 down 0 1.00000 54 ssd 1.74609 osd.54 up 1.00000 1.00000 58 ssd 1.74609 osd.58 up 1.00000 1.00000 59 ssd 1.74609 osd.59 up 1.00000 1.00000 67 ssd 1.74660 osd.67 up 1.00000 1.00000 68 ssd 1.74660 osd.68 down 0 1.00000 90 ssd 1.74660 osd.90 up 1.00000 1.00000 root@ceph6:~# fdisk -l | grep 1920383410176 Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain(a)bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts…>

2 4

CEPH Cluster performance review
by Mosharaf Hossain 13 Nov '23

13 Nov '23

Hello Community Currently, I operate a CEPH Cluster utilizing Ceph Octopus version 1.5.2.7, installed through Ansible. The challenge I'm encountering is that, during scrubbing, OSD latency spikes to 300-600 ms, resulting in sluggish performance for all VMs. Additionally, some OSDs fail during the scrubbing process. In such instances, promptly halting the scrubbing resolves the issue. *Summary of CEPH Version*: CEPH Version Number of Nodes Node Networking OSD OSD Total pools PG Size 15.2.7 12 (6 SSD node + 6 HDD node) All nodes are connected through 10G bonded link, i.e. 10Gx2=20GB for each node. 64 SSD 42 HDD 106 one-ssd 256 active+clean one-hdd 512 active+clean cloudstack.hdd 512 active+clean I intend to enlarge the PG size for the "one-ssd" configuration. Please provide the PG number, and suggest the optimal approach to increase the PG size without causing any slowdown or service disruptions to the VMs. Your expertise and guidance on this matter would be highly valuable, and I'm eager to benefit from the collective knowledge of the Ceph community. Thank you in advance for your time and assistance. I look forward to hearing from you. *CEPH Health status:* root@mon1:~# ceph -s cluster: id: f8096ec7-51db-4557-85e6-57d7fdfe9423 health: HEALTH_WARN nodeep-scrub flag(s) set 656 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph2,mon1,ceph6 (age 4d) mgr: ceph4(active, since 2w), standbys: mon1, ceph3, ceph6, ceph1 mds: cephfs:1 {0=ceph8=up:active} 1 up:standby osd: 107 osds: 105 up (since 3h), 105 in (since 3d) flags nodeep-scrub rgw: 4 daemons active (ceph10.rgw0, ceph7.rgw0, ceph9.rgw0, mon1.rgw0) rgw-nfs: 2 daemons active (ceph7, ceph9) task status: data: pools: 13 pools, 2057 pgs objects: 9.40M objects, 35 TiB usage: 106 TiB used, 154 TiB / 259 TiB avail pgs: 2057 active+clean io: client: 14 MiB/s rd, 30 MiB/s wr, 1.50k op/s rd, 1.53k op/s wr root@ceph1:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 151 TiB 78 TiB 72 TiB 73 TiB 48.04 ssd 110 TiB 78 TiB 32 TiB 32 TiB 29.42 TOTAL 261 TiB 156 TiB 104 TiB 105 TiB 40.19 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 64 3.8 KiB 0 11 KiB 0 23 TiB cephfs_metadata 2 8 228 MiB 79 685 MiB 0 23 TiB .rgw.root 3 32 6.0 KiB 8 1.5 MiB 0 23 TiB default.rgw.control 4 32 0 B 8 0 B 0 23 TiB default.rgw.meta 5 32 12 KiB 48 7.5 MiB 0 23 TiB default.rgw.log 6 32 4.8 KiB 207 6.0 MiB 0 23 TiB default.rgw.buckets.index 7 32 410 MiB 15 1.2 GiB 0 23 TiB default.rgw.buckets.data 8 512 4.6 TiB 1.29M 14 TiB 16.59 23 TiB default.rgw.buckets.non-ec 9 32 1.0 MiB 676 130 MiB 0 23 TiB one-hdd 10 512 9.2 TiB 2.45M 28 TiB 28.69 23 TiB device_health_metrics 11 1 9.5 MiB 113 28 MiB 0 23 TiB one-ssd 12 256 11 TiB 2.88M 32 TiB 31.37 23 TiB cloudstack.hdd 15 512 10 TiB 2.72M 31 TiB 30.94 23 TiB Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain(a)bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts…>

4 4

Automatic triggering of the Ubuntu SRU process, e.g. for the recent 17.2.7 Quincy point release?
by Christian Rohmann 12 Nov '23

12 Nov '23

Hey Yuri, hey ceph-users, first of all, thanks for all your work on developing and maintaining Ceph. I was just wondering if there was any sort of process or trigger to the Ubuntu release team following a point release, for them to also create updated packages. If you look at https://packages.ubuntu.com/jammy-updates/ceph, there still only is 17.2.6 as the current update available. There was an [SRU] bug raised for 17.2.6 (https://bugs.launchpad.net/cloud-archive/+bug/2018929) I now opened a similar one (https://bugs.launchpad.net/cloud-archive/+bug/2043336) hoping I went the right way of triggering the packaging this point release. Even though the Ceph team does not build Quincy packages for Ubuntu 22.04 LTS (Jammy) themselves, it would be nice to still treat it somewhat of as a release channel and to automatically trigger these kind of processes. Regards Christian

1 0

v17.2.7 Quincy released
by Yuri Weinstein 12 Nov '23

12 Nov '23

We're happy to announce the 7th backport release in the Quincy series. https://ceph.io/en/news/blog/2023/v17-2-7-quincy-released/ Notable Changes --------------- * `ceph mgr dump` command now displays the name of the Manager module that registered a RADOS client in the `name` field added to elements of the `active_clients` array. Previously, only the address of a module's RADOS client was shown in the `active_clients` array. * mClock Scheduler: The mClock scheduler (default scheduler in Quincy) has undergone significant usability and design improvements to address the slow backfill issue. Some important changes are: * The 'balanced' profile is set as the default mClock profile because it represents a compromise between prioritizing client IO or recovery IO. Users can then choose either the 'high_client_ops' profile to prioritize client IO or the 'high_recovery_ops' profile to prioritize recovery IO. * QoS parameters including reservation and limit are now specified in terms of a fraction (range: 0.0 to 1.0) of the OSD's IOPS capacity. * The cost parameters (osd_mclock_cost_per_io_usec_* and osd_mclock_cost_per_byte_usec_*) have been removed. The cost of an operation is now determined using the random IOPS and maximum sequential bandwidth capability of the OSD's underlying device. * Degraded object recovery is given higher priority when compared to misplaced object recovery because degraded objects present a data safety issue not present with objects that are merely misplaced. Therefore, backfilling operations with the 'balanced' and 'high_client_ops' mClock profiles may progress slower than what was seen with the 'WeightedPriorityQueue' (WPQ) scheduler. * The QoS allocations in all mClock profiles are optimized based on the above fixes and enhancements. * For more detailed information see: https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ * RGW: S3 multipart uploads using Server-Side Encryption now replicate correctly in multi-site. Previously, the replicas of such objects were corrupted on decryption. A new tool, ``radosgw-admin bucket resync encrypted multipart``, can be used to identify these original multipart uploads. The ``LastModified`` timestamp of any identified object is incremented by 1 nanosecond to cause peer zones to replicate it again. For multi-site deployments that make any use of Server-Side Encryption, we recommended running this command against every bucket in every zone after all zones have upgraded. * CephFS: MDS evicts clients which are not advancing their request tids which causes a large buildup of session metadata resulting in the MDS going read-only due to the RADOS operation exceeding the size threshold. `mds_session_metadata_threshold` config controls the maximum size that a (encoded) session metadata can grow. * CephFS: After recovering a Ceph File System post following the disaster recovery procedure, the recovered files under `lost+found` directory can now be deleted. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: b12291d110049b2f35e32e0de30d70e9a4c060d2

2 1

check_memory_usage() recreation in OSD:tick()
by Suyash Dongre 12 Nov '23

12 Nov '23

Hi guys!This is a function for MDS (src/mds/MDCache.cc) that prints out the rss and other memory stats every couple seconds at debug level 2.I am trying to create a similar function for printing memory and other stats for OSD. (will be adding the code in src/osd/OSD:tick() function),can anyone provide a basic guidelines about what all classes, functions I will need to use and what all will I require for achieving so?Any help would be extremely valuable, Thanks for your time! void MDCache::check_memory_usage() { static MemoryModel mm(g_ceph_context); static MemoryModel::snap last; mm.sample(&last); static MemoryModel::snap baseline = last; // check client caps ceph_assert(CInode::count() == inode_map.size() + snap_inode_map.size() + num_shadow_inodes); double caps_per_inode = 0.0; if (CInode::count()) caps_per_inode = (double)Capability::count() / (double)CInode::count(); dout(2) << "Memory usage: " << " total " << last.get_total() << ", rss " << last.get_rss() << ", heap " << last.get_heap() << ", baseline " << baseline.get_heap() << ", " << num_inodes_with_caps << " / " << CInode::count() << " inodes have caps" << ", " << Capability::count() << " caps, " << caps_per_inode << " caps per inode" << dendl; mds->update_mlogger(); mds->mlogger->set(l_mdm_rss, last.get_rss()); mds->mlogger->set(l_mdm_heap, last.get_heap()); }

1 0

one cephfs volume becomes very slow
by Ben 10 Nov '23

10 Nov '23

Dear cephers, we have a cephfs volume, that will be mounted by many clients with concurrent read/write capability. From time to time, maybe when concurrency goes as high as 100 clients' access, accessing it will become very slow to be useful at all. the cluster has multiple active mds. All disks are hdd. Any ideas to improve this? here is one of mds log during the slow time, others are simillar: {"log":"debug 2023-11-08T07:26:00.114+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for \u003e 5.662970 secs\n","stream":"stderr","time":"2023-11-08T07:26:00.121996282Z"} {"log":"debug 2023-11-08T07:26:00.114+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 5.662970 seconds old, received at 2023-11-08T07:25:54.458863+0000: peer_request:client.12917739:8654334 currently dispatched\n","stream":"stderr","time":"2023-11-08T07:26:00.122016551Z"} {"log":"debug 2023-11-08T07:29:54.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for \u003e 11.900602 secs\n","stream":"stderr","time":"2023-11-08T07:29:54.124567293Z"} {"log":"debug 2023-11-08T07:29:54.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 11.900601 seconds old, received at 2023-11-08T07:29:42.223813+0000: client_request(client.27494331:18564666 getattr pAsLsXsFs #0x70001830366 2023-11-08T07:29:42.219416+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting\n","stream":"stderr","time":"2023-11-08T07:29:54.124589613Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 5 slow requests, 5 included below; oldest blocked for \u003e 17.900670 secs\n","stream":"stderr","time":"2023-11-08T07:30:00.124691442Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 17.900670 seconds old, received at 2023-11-08T07:29:42.223813+0000: client_request(client.27494331:18564666 getattr pAsLsXsFs #0x70001830366 2023-11-08T07:29:42.219416+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting\n","stream":"stderr","time":"2023-11-08T07:30:00.124726772Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.649942 seconds old, received at 2023-11-08T07:29:53.474541+0000: client_request(mds.1:305661 rename #0x70001851b32/91e670f9004ddb237a353b2a9ddc063208f5 #0x649/800019f1da7 caller_uid=0, caller_gid=0{}) currently failed to acquire_locks\n","stream":"stderr","time":"2023-11-08T07:30:00.124731626Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.649864 seconds old, received at 2023-11-08T07:29:53.474619+0000: client_request(mds.1:305662 rename #0x70001851b32/91e670f9004ddb237a353b2a9ddc063208f5 #0x649/800019f1da7 caller_uid=0, caller_gid=0{}) currently requesting remote authpins\n","stream":"stderr","time":"2023-11-08T07:30:00.124734415Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.649719 seconds old, received at 2023-11-08T07:29:53.474764+0000: client_request(client.27497255:25173 getattr pAsLsXsFs #0x800019f1da7 2023-11-08T07:29:53.473182+0000 caller_uid=0, caller_gid=0{}) currently requesting remote authpins\n","stream":"stderr","time":"2023-11-08T07:30:00.124736973Z"} {"log":"debug 2023-11-08T07:30:00.118+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.648454 seconds old, received at 2023-11-08T07:29:53.476029+0000: client_request(mds.1:305663 rename #0x70001851b32/91e670f9004ddb237a353b2a9ddc063208f5 #0x649/800019f1da7 caller_uid=0, caller_gid=0{}) currently requesting remote authpins\n","stream":"stderr","time":"2023-11-08T07:30:00.124739607Z"} {"log":"debug 2023-11-08T07:43:30.127+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 2 slow requests, 2 included below; oldest blocked for \u003e 5.206645 secs\n","stream":"stderr","time":"2023-11-08T07:43:30.133682292Z"} {"log":"debug 2023-11-08T07:43:30.127+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 5.206644 seconds old, received at 2023-11-08T07:43:24.926862+0000: client_request(client.27430891:5371608 mkdir #0x700018317cd/13 2023-11-08T07:43:24.924423+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply\n","stream":"stderr","time":"2023-11-08T07:43:30.133708161Z"} {"log":"debug 2023-11-08T07:43:30.127+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 5.206209 seconds old, received at 2023-11-08T07:43:24.927297+0000: client_request(client.27430891:5371609 create #0x2000d8a535a/2d69440b170ada3dd3670b5f3e2ebe4d75eeb274ff2a8238a23162e95601 2023-11-08T07:43:24.924423+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply\n","stream":"stderr","time":"2023-11-08T07:43:30.133711838Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 5 slow requests, 5 included below; oldest blocked for \u003e 7.332759 secs\n","stream":"stderr","time":"2023-11-08T07:50:18.138195062Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 7.318159 seconds old, received at 2023-11-08T07:50:10.819829+0000: client_request(client.27041435:5802294 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.815700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:18.138221837Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 7.305700 seconds old, received at 2023-11-08T07:50:10.832287+0000: client_request(client.27041435:5802298 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.827700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:18.138225662Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 6.977220 seconds old, received at 2023-11-08T07:50:11.160767+0000: client_request(client.27041435:5802299 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:11.155701+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:18.138228595Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 7.332758 seconds old, received at 2023-11-08T07:50:10.805229+0000: client_request(client.27041435:5802289 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.799700+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting\n","stream":"stderr","time":"2023-11-08T07:50:18.138231253Z"} {"log":"debug 2023-11-08T07:50:18.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 7.327715 seconds old, received at 2023-11-08T07:50:10.810272+0000: client_request(client.27041435:5802290 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.803700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:18.138234251Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 5 slow requests, 5 included below; oldest blocked for \u003e 13.332825 secs\n","stream":"stderr","time":"2023-11-08T07:50:24.138201305Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 13.318225 seconds old, received at 2023-11-08T07:50:10.819829+0000: client_request(client.27041435:5802294 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.815700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:24.138221584Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 13.305766 seconds old, received at 2023-11-08T07:50:10.832287+0000: client_request(client.27041435:5802298 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.827700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:24.13822536Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 12.977286 seconds old, received at 2023-11-08T07:50:11.160767+0000: client_request(client.27041435:5802299 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:11.155701+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:24.138242952Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 13.332825 seconds old, received at 2023-11-08T07:50:10.805229+0000: client_request(client.27041435:5802289 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.799700+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting\n","stream":"stderr","time":"2023-11-08T07:50:24.138245661Z"} {"log":"debug 2023-11-08T07:50:24.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 13.327781 seconds old, received at 2023-11-08T07:50:10.810272+0000: client_request(client.27041435:5802290 getattr pAsLsXsFs #0x7000182f285 2023-11-08T07:50:10.803700+0000 caller_uid=0, caller_gid=0{}) currently dispatched\n","stream":"stderr","time":"2023-11-08T07:50:24.13824822Z"} {"log":"debug 2023-11-08T07:53:54.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for \u003e 5.641663 secs\n","stream":"stderr","time":"2023-11-08T07:53:54.140575892Z"} {"log":"debug 2023-11-08T07:53:54.134+0000 7f190b014700 0 log_channel(cluster) log [WRN] : slow request 5.641662 seconds old, received at 2023-11-08T07:53:48.498707+0000: client_request(client.12072730:30349041 rename #0x70001831e1d/ec603fcfb5f191c00efd5f87d2fb96cbd04a40928b9d1f11744d7ef6d742 #0x7000182f285/e9e5e277 2023-11-08T07:53:48.497133+0000 caller_uid=0, caller_gid=0{}) currently acquired locks\n","stream":"stderr","time":"2023-11-08T07:53:54.140612721Z"}

2 3

Permanent KeyError: 'TYPE' ->17.2.7: return self.blkid_api['TYPE'] == 'part'
by Harry G Coin 10 Nov '23

10 Nov '23

These repeat for every host, only after upgrading from prev release Quincy to 17.2.7. As a result, the cluster is always warned, never indicates healthy. root@noc1:~# ceph health detail HEALTH_WARN failed to probe daemons or devices [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host sysmon1 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/mon.sysmon1/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha2 56:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e -e NODE_NAME=sysmon1 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/4067126d-01cb-40af-824a-881c130140f8:/var /run/ceph:z -v /var/log/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/log/ceph:z -v /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lv m -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpl1e27bun:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e inventory --format=json-pretty --filter-for-batch /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 60, in main /usr/bin/docker: stderr list_all=self.args.list_all)) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 50, in __init__ /usr/bin/docker: stderr sys_info.devices.keys()] /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 49, in <listcomp> /usr/bin/docker: stderr all_devices_vgs=all_devices_vgs) for k in /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 147, in __init__ /usr/bin/docker: stderr self.available_lvm, self.rejected_reasons_lvm = self._check_lvm_reject_reasons() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 646, in _check_lvm_reject_reasons /usr/bin/docker: stderr rejected.extend(self._check_generic_reject_reasons()) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 601, in _check_generic_reject_reasons /usr/bin/docker: stderr if self.is_acceptable_device: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 502, in is_acceptable_device /usr/bin/docker: stderr return self.is_device or self.is_partition or self.is_lv /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 482, in is_partition /usr/bin/docker: stderr return self.blkid_api['TYPE'] == 'part' /usr/bin/docker: stderr KeyError: 'TYPE' Traceback (most recent call last): File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9679, in <module> main() File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9667, in main r = ctx.func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2116, in _infer_config return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2032, in _infer_fsid return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2144, in _infer_image return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2019, in _validate_fsid return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 6272, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 1807, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@ sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e -e NODE_NAME=sysmon1 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/4067126d-01cb-40af-824a-881c130140f8: /var/run/ceph:z -v /var/log/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/log/ceph:z -v /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/ru n/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpl1e27bun:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e inventory --format=json-pretty --filter-for-b atch host noc3 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/mon.noc3/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha2 56:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e -e NODE_NAME=noc3 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/ru n/ceph:z -v /var/log/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/log/ceph:z -v /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm - v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpjjmsgag7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e inventory --format=json-pretty --filter-for-batch /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 60, in main /usr/bin/docker: stderr list_all=self.args.list_all)) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 50, in __init__ /usr/bin/docker: stderr sys_info.devices.keys()] /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 49, in <listcomp> /usr/bin/docker: stderr all_devices_vgs=all_devices_vgs) for k in /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 147, in __init__ /usr/bin/docker: stderr self.available_lvm, self.rejected_reasons_lvm = self._check_lvm_reject_reasons() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 646, in _check_lvm_reject_reasons /usr/bin/docker: stderr rejected.extend(self._check_generic_reject_reasons()) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 601, in _check_generic_reject_reasons /usr/bin/docker: stderr if self.is_acceptable_device: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 502, in is_acceptable_device /usr/bin/docker: stderr return self.is_device or self.is_partition or self.is_lv /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 482, in is_partition /usr/bin/docker: stderr return self.blkid_api['TYPE'] == 'part' /usr/bin/docker: stderr KeyError: 'TYPE' Traceback (most recent call last): File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9679, in <module> main() File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 9667, in main r = ctx.func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2116, in _infer_config return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2032, in _infer_fsid return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2144, in _infer_image return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 2019, in _validate_fsid return func(ctx) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 6272, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) File "/var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b", line 1807, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@ sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e -e NODE_NAME=noc3 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/4067126d-01cb-40af-824a-881c130140f8:/va r/run/ceph:z -v /var/log/ceph/4067126d-01cb-40af-824a-881c130140f8:/var/log/ceph:z -v /var/lib/ceph/4067126d-01cb-40af-824a-881c130140f8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/l vm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpjjmsgag7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:92e8fa7d8ca17a7a5bbfde6e596fdfecc8e165fcb94d86493f4e6c7b1f326e4e inventory --format=json-pretty --filter-for-batc h ....

2 2

Redeploy ceph orch OSDs after reboot, but don't mark as 'unmanaged'
by Janek Bevendorff 10 Nov '23

10 Nov '23

Hi, We have our cluster RAM-booted, so we start from a clean slate after every reboot. That means I need to redeploy all OSD daemons as well. At the moment, I run cephadm deploy via Salt on the rebooted node, which brings the deployed OSDs back up, but the problem with this is that the deployed OSD shows up as 'unmanaged' in ceph orch ps afterwards. I could simply skip the cephadm call and wait for the Ceph orchestrator to reconcile and auto-activate the disks, but that can take up to 15 minutes, which is unacceptable. Running ceph cephadm osd activate is not an option either, since I don't have the admin keyring deployed on the OSD hosts (I could do that, but I don't want to). How can I manually activate the OSDs after a reboot and hand over control to the Ceph orchestrator afterwards? I checked the deployments in /var/lib/ceph/<FSID>, but the only difference I found between my manual cephadm deployment and what ceph orch does is that the device links to /dev/mapper/ceph--... instead of /dev/ceph-... Any hints appreciated! Janek

2 6

IO stalls when primary OSD device blocks in 17.2.6
by Daniel Schreiber 10 Nov '23

10 Nov '23

Dear cephers, we are sometimes observing stalling IO on our ceph 17.2.6 cluster when the backing device for the primary OSD of a PG fails and seems to block read IO to objects from that pg. If I set the OSD with the broken device to down, the IO continues. Setting the OSD to down is not sufficient. The cluster is running on Debian 11, the pool is an erasure coded cephfs data pool. The OSD has a HDD data device and an SSD db device. The data devices is the one which failed and was blocking IO. The OSD was reporting slow ops and short time after that smartd notified about unreadable sectors. Has anyone seen such behaviour? Are there some tweaks that I missed? Kind regards, Daniel -- Daniel Schreiber Facharbeitsgruppe Systemsoftware Universitaetsrechenzentrum Technische Universität Chemnitz Straße der Nationen 62 (Raum B303) 09111 Chemnitz Germany Tel: +49 371 531 35444 Fax: +49 371 531 835444

2 1

Re: Reinitialize rgw garbage collector
by Pierre GINDRAUD 10 Nov '23

10 Nov '23

Hello Michael, Did you receive any help on this ? We are issuing the same problem without solution for now. Regard -- Pierre

1 0

Help needed with Grafana password
by Sake Ceph 10 Nov '23

10 Nov '23

2 10

mds hit find_exports balancer runs too long
by zxcs 10 Nov '23

10 Nov '23

Hi, Experts, we have a CephFS cluster running with 16.2.*, and enable multi active mds, found somehow mds complain some info as below: mds.*.bal find_exports balancer runs too long and we already set below config mds_bal_interval = 30 mds_bal_sample_interval = 12 and then we can see slow mds request from `ceph -s`. our question is : why ceph mds complain this and how can we prevent this problem from happening again? Thanks a ton, xiong

1 0

Ceph Dashboard - Community News Sticker [Feedback]
by Nizamudeen A 10 Nov '23

10 Nov '23

Hello, We wanted to get some feedback on one of the features that we are planning to bring in for upcoming releases. On the Ceph GUI, we thought it could be interesting to show information regarding the community events, ceph release information (Release notes and changelogs) and maybe even notify about new blog post releases and also inform regarding the community group meetings. There would be options to subscribe to the events that you want to get notified. Before proceeding with its implementation, we thought it'd be good to get some community feedback around it. So please let us know what you think (the goods and the bads). Regards, -- Nizamudeen A Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

7 7

MDS stuck in rejoin
by Frank Schilder 10 Nov '23

10 Nov '23

Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush tid". I looked at the client and there was nothing going on, so I rebooted it. After the client was back, the message was still there. To clean this up I failed the MDS. Unfortunately, the MDS that took over is remained stuck in rejoin without doing anything. All that happened in the log was: [root@ceph-10 ceph]# tail -f ceph-mds.ceph-10.log 2023-07-20T15:54:29.147+0200 7fedb9c9f700 1 mds.2.896604 rejoin_start 2023-07-20T15:54:29.161+0200 7fedb9c9f700 1 mds.2.896604 rejoin_joint_start 2023-07-20T15:55:28.005+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896614 from mon.4 2023-07-20T15:56:00.278+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896615 from mon.4 [...] 2023-07-20T16:02:54.935+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896653 from mon.4 2023-07-20T16:03:07.276+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896654 from mon.4 After some time I decided to give another fail a try and, this time, the replacement daemon went to active state really fast. If I have a message like the above, what is the clean way of getting the client clean again (version: 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable))? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 16

HDD cache
by Peter 09 Nov '23

09 Nov '23

Hi All, I note that HDD cluster commit delay improves after i turn off HDD cache. However, i also note that not all HDDs are able to turn off the cache. special I found that two HDD with same model number, one can turn off, anther doesn't. i guess i have my system config or something different setting with two HDDs. Below is my command to turn off the HDD cache. root@lahost008:~# sdparm --set WCE=0 /dev/sdb /dev/sdb: ATA ST10000NE000-3AP EN01 root@lahost008:~# cat /sys/block/sdb/queue/write_cache write through I also tried sdparm to run "sdparm --set WCE=0 /dev/sdb" I got the same result. Anyone experienced this can advise? Thanks a lot

4 4

Memory footprint of increased PG number
by Nicola Mori 09 Nov '23

09 Nov '23

Dear Ceph user, I'm wondering how much an increase of PG number would impact on the memory occupancy of OSD daemons. In my cluster I currently have 512 PGs and I would like to increase it to 1024 to mitigate some disk occupancy issues, but having machines with low amount of memory (down to 24 GB for 16 OSDs) I fear this could kill my cluster. Is it possible to evaluate the relative increase in OSD memory footprint when doubling the number of PGs (hopefully not a linear scaling)? Or is there a way to experiment without crashing everything? Thank you.

2 2

High iowait when using Ceph NVME
by Huy Nguyen 09 Nov '23

09 Nov '23

Hi, Currently, I'm testing Ceph v17.2.7 with NVMe. When mapping an rbd image to physical compute host, "fio bs=4k iodepth=128 randwrite" give 150k IOPS. I have a VM that located within the compute host, and the fio give ~40k IOPS and with 50 %iowait. I know there is a bottleneck, I'm not sure where it is though I have tried to enabled iothread in virsh but nothing change. Anyone have any ideal? Thanks in advance.

1 0

Found unknown daemon type ceph-exporter on host after upgrade to 17.2.7
by Dmitry Melekhov 09 Nov '23

09 Nov '23

Hello! I see [WRN] Found unknown daemon type ceph-exporter on host for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard and cephadm ['--image', 'quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb', '--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter in cephadm.log There were no such messages before. I guess this will not create any problem, but is there any way to fix this?

4 9

Crush map & rule
by Albert Shih 09 Nov '23

09 Nov '23

Hi everyone, I'm totally newbie with ceph, so sorry if I'm asking some stupid question. I'm trying to understand how the crush map & rule work, my goal is to have two groups of 3 servers, so I'm using “row” bucket ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 59.38367 root default -15 59.38367 zone City -17 29.69183 row primary -3 9.89728 host server1 0 ssd 3.49309 osd.0 up 1.00000 1.00000 1 ssd 1.74660 osd.1 up 1.00000 1.00000 2 ssd 1.74660 osd.2 up 1.00000 1.00000 3 ssd 2.91100 osd.3 up 1.00000 1.00000 -5 9.89728 host server2 4 ssd 1.74660 osd.4 up 1.00000 1.00000 5 ssd 1.74660 osd.5 up 1.00000 1.00000 6 ssd 2.91100 osd.6 up 1.00000 1.00000 7 ssd 3.49309 osd.7 up 1.00000 1.00000 -7 9.89728 host server3 8 ssd 3.49309 osd.8 up 1.00000 1.00000 9 ssd 1.74660 osd.9 up 1.00000 1.00000 10 ssd 2.91100 osd.10 up 1.00000 1.00000 11 ssd 1.74660 osd.11 up 1.00000 1.00000 -19 29.69183 row secondary -9 9.89728 host server4 12 ssd 1.74660 osd.12 up 1.00000 1.00000 13 ssd 1.74660 osd.13 up 1.00000 1.00000 14 ssd 3.49309 osd.14 up 1.00000 1.00000 15 ssd 2.91100 osd.15 up 1.00000 1.00000 -11 9.89728 host server5 16 ssd 1.74660 osd.16 up 1.00000 1.00000 17 ssd 1.74660 osd.17 up 1.00000 1.00000 18 ssd 3.49309 osd.18 up 1.00000 1.00000 19 ssd 2.91100 osd.19 up 1.00000 1.00000 -13 9.89728 host server6 20 ssd 1.74660 osd.20 up 1.00000 1.00000 21 ssd 1.74660 osd.21 up 1.00000 1.00000 22 ssd 2.91100 osd.22 up 1.00000 1.00000 and I want to create a some rules, first I like to have a rule «replica» (over host) inside the «row» primary a rule «erasure» (over host) inside the «row» primary but also two crush rule between primary/secondary, meaning I like to have a replica (with only 1 copy of course) of pool from “row” primary to secondary. How can I achieve that ? Regards -- Albert SHIH 🦫 🐸 mer. 08 nov. 2023 18:37:54 CET

2 3

Question about PG mgr/balancer/crush_compat_metrics
by bryansoong21＠gmail.com 08 Nov '23

08 Nov '23

Hello, We are using a Ceph Pacific (16.2.10) cluster and enabled the balancer module, but the usage of some OSDs keeps growing and reached up to mon_osd_nearfull_ratio, which we use 85% by default, and we think the balancer module should do some balancer work. So I checked our balancer configuration and found that our "crush_compat_metrics" is set to "pgs,objects,bytes", and this three values are used in src.pybind.mgr.balancer.module.Module.calc_eval. However, when doing the actual balance task, only the first key is used to do the auto balance, in src.pybind.mgr.balancer.module.Module.do_crush_compat: metrics = self.get_module_option('crush_compat_metrics').split(',') key = metrics[0] # balancing using the first score metric My concern is, any reason why we calculate the balancing using the three items but only do the balance using the first one? Thanks.

2 1

Ceph Leadership Team Weekly Meeting Minutes 2023-11-08
by Patrick Donnelly 08 Nov '23

08 Nov '23

Hello all, Here are the minutes from today's meeting. - New time for CDM APAC to increase participation - 9.30 - 11.30 pm PT seems like the most popular based on https://doodle.com/meeting/participate/id/aM9XGZ3a/vote - One more week for more feedback; please ask more APAC folks to suggest their preferred times. - [Ernesto] Revamp Ansible/Ceph-Ansible for non-containerized users? - open nebula / proxmox - solicit maintainers for ceph-ansible on the ML - 18.2.1 - yuri: approval email sent out a few days ago; waiting on some approvals - Blocker: - https://tracker.ceph.com/issues/63391 - lab upgrades (Laura will help Yuri coordinate) - Next Pacific release being worked on in background by Yuri. - https://pad.ceph.com/p/pacific_16.2.15 - Try v16.2.15 milestone to help prune PRs - https://github.com/ceph/ceph/milestone/17 - [Nizam] Ceph News Ticker - Ceph Dashboard - Notify when new release is available (display changelogs) - Display important ceph events - CVEs, critical bug fixes - Maybe newly added blog posts or informations regarding the upcoming group meetings? - User + Dev meeting next week - Topics include migration between EC profiles and challenges related to RGW zone replication - Casey can attend end of meeting - open nebula folks planning to do webinar; looking for speakers -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

ceph storage pool error
by necoe0147＠gmail.com 08 Nov '23

08 Nov '23

Ceph 3 clusters are running and the 3rd cluster gave an error, it is currently offline. I want to get all the remaining data in 2 clusters. Instead of fixing ceph, I just want to save the data. How can I access this data and connect to the pool? Can you help me?1 and 2 clusters are working. I want to view my data from them and then transfer them to another place. How can I do this? I have never used Ceph before.

2 1

Ceph OSD reported Slow operations
by prabhav＠cdac.in 08 Nov '23

08 Nov '23

In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000 After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup

3 3

list cephfs dirfrags
by Ben 08 Nov '23

08 Nov '23

Hi, I used this but all returns "directory inode not in cache" ceph tell mds.* dirfrag ls path I would like to pin some subdirs to a rank after dynamic subtree partitioning. Before that, I need to know where are they exactly Thank you, Ben

2 2

Help needed with Grafana password
by Sake Ceph 08 Nov '23

08 Nov '23

I configured a password for Grafana because I want to use Loki. I used the spec parameter initial_admin_password and this works fine for a staging environment, where I never tried to used Grafana with a password for Loki. Using the username admin with the configured password gives a credentials error on environment where I tried to use Grafana with Loki in the past (with 17.2.6 of Ceph/cephadm). I changed the password in the past within Grafana, but how can I overwrite this now? Or is there a way to cleanup all Grafana files? Best regards, Sake

1 0

owner locked out of bucket via bucket policy
by Wesley Dillingham 08 Nov '23

08 Nov '23

I have a bucket which got injected with bucket policy which locks the bucket even to the bucket owner. The bucket now cannot be accessed (even get its info or delete bucket policy does not work) I have looked in the radosgw-admin command for a way to delete a bucket policy but do not see anything. I presume I will need to somehow remove the bucket policy from however it is stored in the bucket metadata / omap etc. If anyone can point me in the right direction on that I would appreciate it. Thanks Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

4 15

Seagate Exos power settings - any experiences at your sites?
by Alex Gorbachev 08 Nov '23

08 Nov '23

We have been seeing some odd behavior with scrubbing (very slow) and OSD warnings on a couple of new clusters. A bit of research turned up this: https://www.reddit.com/r/truenas/comments/p1ebnf/seagate_exos_load_cyclingi… We've installed the tool from https://github.com/Seagate/openSeaChest and disabled EPC power features similar to: openSeaChest_PowerControl --scan|grep ST|awk '{print $2}'|xargs -I {} openSeaChest_PowerControl -d {} --EPCfeature disable Things seem to be better now on those two clusters. Has anyone seen anything similar? This would seem to be a huge issue if all defaults on Exos are wrong (stop-and-go on all Ceph/ZFS workloads). -- Best regards, Alex Gorbachev -- Intelligent Systems Services Inc. http://www.iss-integration.com https://www.linkedin.com/in/alex-gorbachev-iss/

2 1

100.00 Usage for ssd-pool (maybe after: ceph osd crush move .. root=default)
by Denny Fuchs 08 Nov '23

08 Nov '23

Hello, we upgraded to Quincy and tried to remove an obsolete part: In the beginning of Ceph, there where no device classes and we created rules, to split them into hdd and ssd on one of our datacenters. https://www.sebastien-han.fr/blog/2014/08/25/ceph-mix-sata-and-ssd-within-t… So we had different "roots" for SSD and HDD. Two weeks ago .. we tried to move the hosts to the root=default and checked .. what happens .. nothing .. all was fine and working. But we did not checked the "ceph df": ========== root@fc-r02-ceph-osd-01:[~]: ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -14 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root sata -18 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter fc-sata -16 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - rack r02-sata -13 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root ssds -17 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter fc-ssds -15 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - rack r02-ssds -1 23.99060 - 23 TiB 13 TiB 12 TiB 6.0 GiB 32 GiB 11 TiB 54.17 1.00 - root default -6 4.00145 - 3.9 TiB 2.1 TiB 2.1 TiB 2.1 MiB 7.2 GiB 1.7 TiB 54.87 1.01 - host fc-r02-ceph-osd-01 0 ssd 0.45470 1.00000 447 GiB 236 GiB 235 GiB 236 KiB 794 MiB 211 GiB 52.80 0.97 119 up osd.0 1 ssd 0.45470 1.00000 447 GiB 222 GiB 221 GiB 239 KiB 808 MiB 225 GiB 49.67 0.92 108 up osd.1 2 ssd 0.45470 1.00000 447 GiB 245 GiB 244 GiB 254 KiB 819 MiB 202 GiB 54.85 1.01 118 up osd.2 3 ssd 0.45470 1.00000 447 GiB 276 GiB 276 GiB 288 KiB 903 MiB 171 GiB 61.83 1.14 135 up osd.3 4 ssd 0.45470 1.00000 447 GiB 268 GiB 267 GiB 272 KiB 913 MiB 180 GiB 59.85 1.10 132 up osd.4 5 ssd 0.45470 1.00000 447 GiB 204 GiB 203 GiB 181 KiB 684 MiB 243 GiB 45.56 0.84 108 up osd.5 41 ssd 0.36388 1.00000 373 GiB 211 GiB 210 GiB 207 KiB 818 MiB 161 GiB 56.69 1.05 104 up osd.41 42 ssd 0.45470 1.00000 447 GiB 220 GiB 219 GiB 214 KiB 791 MiB 227 GiB 49.26 0.91 107 up osd.42 48 ssd 0.45470 1.00000 447 GiB 284 GiB 284 GiB 281 KiB 864 MiB 163 GiB 63.62 1.17 139 up osd.48 -2 3.98335 - 3.9 TiB 2.1 TiB 2.1 TiB 1.0 GiB 5.0 GiB 1.7 TiB 54.82 1.01 - host fc-r02-ceph-osd-02 36 nvme 0.36388 1.00000 373 GiB 239 GiB 238 GiB 163 MiB 460 MiB 134 GiB 64.10 1.18 127 up osd.36 6 ssd 0.45470 1.00000 447 GiB 247 GiB 246 GiB 114 MiB 585 MiB 200 GiB 55.20 1.02 121 up osd.6 7 ssd 0.45470 1.00000 447 GiB 260 GiB 259 GiB 158 MiB 590 MiB 187 GiB 58.19 1.07 126 up osd.7 8 ssd 0.45470 1.00000 447 GiB 196 GiB 195 GiB 165 MiB 471 MiB 251 GiB 43.85 0.81 101 up osd.8 9 ssd 0.45470 1.00000 447 GiB 203 GiB 202 GiB 168 MiB 407 MiB 244 GiB 45.34 0.84 104 up osd.9 10 ssd 0.43660 1.00000 447 GiB 284 GiB 283 GiB 287 KiB 777 MiB 163 GiB 63.49 1.17 142 up osd.10 29 ssd 0.45470 1.00000 447 GiB 241 GiB 240 GiB 147 MiB 492 MiB 206 GiB 53.93 1.00 124 up osd.29 43 ssd 0.45470 1.00000 447 GiB 257 GiB 256 GiB 151 MiB 509 MiB 190 GiB 57.48 1.06 131 up osd.43 49 ssd 0.45470 1.00000 447 GiB 239 GiB 238 GiB 242 KiB 820 MiB 209 GiB 53.35 0.98 123 up osd.49 -5 4.00145 - 3.9 TiB 2.1 TiB 2.1 TiB 1.3 GiB 4.9 GiB 1.7 TiB 55.41 1.02 - host fc-r02-ceph-osd-03 40 nvme 0.36388 1.00000 373 GiB 236 GiB 235 GiB 156 MiB 469 MiB 137 GiB 63.26 1.17 119 up osd.40 11 ssd 0.45470 1.00000 447 GiB 244 GiB 244 GiB 187 MiB 602 MiB 203 GiB 54.68 1.01 123 up osd.11 12 ssd 0.45470 1.00000 447 GiB 235 GiB 235 GiB 163 MiB 496 MiB 212 GiB 52.65 0.97 122 up osd.12 13 ssd 0.45470 1.00000 447 GiB 236 GiB 235 GiB 114 MiB 594 MiB 211 GiB 52.79 0.97 124 up osd.13 14 ssd 0.45470 1.00000 447 GiB 259 GiB 258 GiB 145 MiB 475 MiB 188 GiB 57.87 1.07 126 up osd.14 15 ssd 0.45470 1.00000 447 GiB 267 GiB 266 GiB 181 MiB 662 MiB 180 GiB 59.73 1.10 137 up osd.15 16 ssd 0.45470 1.00000 447 GiB 256 GiB 255 GiB 183 MiB 478 MiB 191 GiB 57.19 1.06 134 up osd.16 44 ssd 0.45470 1.00000 447 GiB 229 GiB 228 GiB 188 MiB 579 MiB 219 GiB 51.12 0.94 117 up osd.44 50 ssd 0.45470 1.00000 447 GiB 227 GiB 226 GiB 215 KiB 688 MiB 220 GiB 50.72 0.94 113 up osd.50 -9 4.00145 - 3.9 TiB 2.0 TiB 2.0 TiB 1.2 GiB 4.8 GiB 1.8 TiB 52.17 0.96 - host fc-r02-ceph-osd-04 37 nvme 0.36388 1.00000 373 GiB 198 GiB 198 GiB 166 MiB 561 MiB 174 GiB 53.23 0.98 103 up osd.37 30 ssd 0.45470 1.00000 447 GiB 246 GiB 245 GiB 144 MiB 494 MiB 201 GiB 55.05 1.02 128 up osd.30 31 ssd 0.45470 1.00000 447 GiB 250 GiB 249 GiB 159 MiB 598 MiB 197 GiB 55.86 1.03 128 up osd.31 32 ssd 0.45470 1.00000 447 GiB 238 GiB 238 GiB 173 MiB 459 MiB 209 GiB 53.30 0.98 124 up osd.32 33 ssd 0.45470 1.00000 447 GiB 209 GiB 208 GiB 134 MiB 417 MiB 238 GiB 46.67 0.86 107 up osd.33 34 ssd 0.45470 1.00000 447 GiB 231 GiB 230 GiB 132 MiB 588 MiB 216 GiB 51.71 0.95 115 up osd.34 35 ssd 0.45470 1.00000 447 GiB 223 GiB 223 GiB 135 MiB 537 MiB 224 GiB 49.95 0.92 112 up osd.35 45 ssd 0.45470 1.00000 447 GiB 249 GiB 248 GiB 152 MiB 571 MiB 198 GiB 55.64 1.03 126 up osd.45 51 ssd 0.45470 1.00000 447 GiB 216 GiB 215 GiB 212 KiB 726 MiB 231 GiB 48.34 0.89 110 up osd.51 -11 4.00145 - 3.9 TiB 2.1 TiB 2.1 TiB 1.2 GiB 5.1 GiB 1.8 TiB 53.77 0.99 - host fc-r02-ceph-osd-05 38 nvme 0.36388 1.00000 373 GiB 193 GiB 192 GiB 104 MiB 561 MiB 180 GiB 51.77 0.96 99 up osd.38 17 ssd 0.45470 1.00000 447 GiB 216 GiB 216 GiB 184 MiB 440 MiB 231 GiB 48.40 0.89 114 up osd.17 18 ssd 0.45470 1.00000 447 GiB 255 GiB 255 GiB 178 MiB 540 MiB 192 GiB 57.12 1.05 132 up osd.18 19 ssd 0.45470 1.00000 447 GiB 236 GiB 235 GiB 143 MiB 571 MiB 211 GiB 52.78 0.97 121 up osd.19 20 ssd 0.45470 1.00000 447 GiB 205 GiB 204 GiB 136 MiB 410 MiB 242 GiB 45.79 0.85 105 up osd.20 21 ssd 0.45470 1.00000 447 GiB 233 GiB 232 GiB 149 MiB 459 MiB 214 GiB 52.02 0.96 119 up osd.21 22 ssd 0.45470 1.00000 447 GiB 283 GiB 282 GiB 174 MiB 708 MiB 164 GiB 63.33 1.17 143 up osd.22 46 ssd 0.45470 1.00000 447 GiB 230 GiB 229 GiB 197 MiB 493 MiB 217 GiB 51.41 0.95 112 up osd.46 52 ssd 0.45470 1.00000 447 GiB 273 GiB 272 GiB 309 KiB 1021 MiB 174 GiB 60.99 1.13 141 up osd.52 -25 4.00145 - 3.9 TiB 2.1 TiB 2.1 TiB 1.3 GiB 5.1 GiB 1.8 TiB 53.95 1.00 - host fc-r02-ceph-osd-06 39 nvme 0.36388 1.00000 373 GiB 211 GiB 210 GiB 126 MiB 435 MiB 162 GiB 56.56 1.04 106 up osd.39 23 ssd 0.45470 1.00000 447 GiB 221 GiB 221 GiB 192 MiB 565 MiB 226 GiB 49.51 0.91 120 up osd.23 24 ssd 0.45470 1.00000 447 GiB 270 GiB 270 GiB 204 MiB 540 MiB 177 GiB 60.51 1.12 140 up osd.24 25 ssd 0.45470 1.00000 447 GiB 224 GiB 223 GiB 119 MiB 514 MiB 223 GiB 50.07 0.92 113 up osd.25 26 ssd 0.45470 1.00000 447 GiB 249 GiB 249 GiB 182 MiB 603 MiB 198 GiB 55.77 1.03 127 up osd.26 27 ssd 0.45470 1.00000 447 GiB 229 GiB 228 GiB 206 MiB 495 MiB 218 GiB 51.17 0.94 114 up osd.27 28 ssd 0.45470 1.00000 447 GiB 231 GiB 230 GiB 147 MiB 629 MiB 216 GiB 51.68 0.95 119 up osd.28 47 ssd 0.45470 1.00000 447 GiB 237 GiB 236 GiB 170 MiB 586 MiB 210 GiB 53.00 0.98 131 up osd.47 53 ssd 0.45470 1.00000 447 GiB 258 GiB 257 GiB 263 KiB 829 MiB 189 GiB 57.69 1.07 138 up osd.53 TOTAL 23 TiB 13 TiB 12 TiB 6.0 GiB 32 GiB 11 TiB 54.17 MIN/MAX VAR: 0.81/1.18 STDDEV: 5.03 ========== Then .. today .. I saw then (via Proxmox add disk to a VM), that the "ssd-pool" has 100% usage, what could not be true. You can see it also in the output: ========================== root@fc-r02-ceph-osd-01:[~]: ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED nvme 1.8 TiB 786 GiB 1.1 TiB 1.1 TiB 57.79 ssd 21 TiB 9.8 TiB 11 TiB 11 TiB 53.86 TOTAL 23 TiB 11 TiB 13 TiB 13 TiB 54.17 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL ssd-pool 1 2048 4.2 TiB 1.14M 12 TiB 100.00 0 B db-pool 4 128 50 MiB 3 151 MiB 100.00 0 B .mgr 5 1 43 MiB 12 130 MiB 0 2.4 TiB ========================== So we really missed something: ========================== root@fc-r02-ceph-osd-01:[~]: ceph osd pool get ssd-pool crush_rule crush_rule: fc-r02-ssdpool root@fc-r02-ceph-osd-01:[~]: ceph osd pool get db-pool crush_rule crush_rule: fc-r02-ssdpool ========================== Below is the (pretty old) crushmap. So, which steps are needed, to get it right ? And will it be causing rebalancing (so better to take the datacenter offline?) ? ========================== root@fc-r02-ceph-osd-01:[~]: ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "fc-r02-ssdpool", "type": 1, "steps": [ { "op": "take", "item": -15, "item_name": "r02-ssds" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "fc-r02-satapool", "type": 1, "steps": [ { "op": "take", "item": -16, "item_name": "r02-sata" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 3, "rule_name": "fc-r02-ssd", "type": 1, "steps": [ { "op": "take", "item": -4, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ============================================ cu denny

2 5

Radosgw object stat olh object attrs what does it mean.
by Selcuk Gultekin 08 Nov '23

08 Nov '23

I'd like to discuss the questions I should ask to understand the values under the 'attrs' of an object in the following JSON data structure and evaluate the health of these objects: I have a sample json output, can you comment on the object state here? { "name": "$image.name", "size": 0, "tag": "", "attrs": { "user.rgw.manifest": "", "user.rgw.olh.idtag": "$tag.uuid", "user.rgw.olh.info": "\u0001\u0001�", "user.rgw.olh.ver": "4" } } What is the purpose of this fields? "user.rgw.manifest" "user.rgw.olh.idtag" "user.rgw.olh.info" "user.rgw.olh.ver" What does the empty value "", signify in the context of the object? How does the absence of value in this field affect the object's health? How is the content of this field generated? (For example, what does the "$tag" value represent?) What is the function of this field? What information does the content of this field carry about the object's status? What does the content of this field signify? (For instance, what does "4" represent?) Does this field represent the object's version? What are the distinguishing features that set this object apart from previous versions?

1 0

6.5 CephFS client - ceph_cap_reclaim_work [ceph] / ceph_con_workfn [libceph] hogged CPU
by Stefan Kooman 08 Nov '23

08 Nov '23

Hi, Since the 6.5 kernel addressed the issue with regards to regression in the readahead handling code... we went ahead and installed this kernel for a couple of mail / web clusters (Ubuntu 6.5.1-060501-generic #202309020842 SMP PREEMPT_DYNAMIC Sat Sep 2 08:48:34 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux). Since then we occasionally see the following being logged by the kernel: [Sun Sep 10 07:19:00 2023] workqueue: delayed_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND [Sun Sep 10 08:41:24 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND [Sun Sep 10 11:05:55 2023] workqueue: delayed_work [ceph] hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND [Sun Sep 10 12:54:38 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND [Sun Sep 10 19:06:37 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND [Mon Sep 11 10:53:33 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND [Tue Sep 12 10:14:03 2023] workqueue: ceph_con_workfn [libceph] hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND [Tue Sep 12 11:14:33 2023] workqueue: ceph_cap_reclaim_work [ceph] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND We wonder if this is a new phenomenon, or that it's rather logged in the new kernel and it was not before. However, we have hit a few OOM situations since we switched to the new kernel because of ceph_cap_reclaim_work events (OOM is because Apache threads keep piling up as it cannot access CephFS). We then also see MDS slow ops reported. This might be related to a backup job that is running on a backup server. We did not observe this behavior on 5.12.19 kernel. Ceph cluster is on 16.2.11 currently. Anyone has some insight on this? Thanks, Stefan

3 12

how to disable ceph version check?
by zxcs 08 Nov '23

08 Nov '23

Hi, Experts, we have a ceph cluster report HEALTH_ERR due to multiple old versions. health: HEALTH_ERR There are daemons running multiple old versions of ceph after run `ceph version`, we see three ceph versions in {16.2.*} , these daemons are ceph osd. our question is: how to stop this version check , we cannot upgrade all old daemon. Thanks, Xiong

2 1

pool(s) do not have an application enabled after upgrade ti 17.2.7
by Dmitry Melekhov 08 Nov '23

08 Nov '23

Hello! I'm very new to ceph ,s orry I'm asking extremely basic questions. I just upgraded 17.2.6 to 17.2.7 and got warning: 2 pool(s) do not have an application enabled These pools are 5 cephfs.cephfs.meta 6 cephfs.cephfs.data I don't remember why and how I created them, I just followed some instruction... And don't remember their state before upgrade :-( And I see in dashboard 0 bytes is used in both pools. But I have two other pools 3 cephfs_data 4 cephfs_metadata which are in use by cephfs: ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] and really have data in them. Could you tell me, can I just remove these two pools without application, if everything works , i.e. cephfs is mounted and accessible? Thank you!

2 6

OSD fails to start after 17.2.6 to 17.2.7 update
by Matthew Booth 07 Nov '23

07 Nov '23

Firstly I'm rolling out a rook update from v1.12.2 to v1.12.7 (latest stable) and ceph from 17.2.6 to 17.2.7 at the same time. I mention this in case the problem is actually caused by rook rather than ceph. It looks like ceph to my uninitiated eyes, though. The update just started bumping my OSDs and the first one fails in the 'activate' init container. The complete logs for this container are: + OSD_ID=5 + CEPH_FSID=<redacted> + OSD_UUID=<redacted> + OSD_STORE_FLAG=--bluestore + OSD_DATA_DIR=/var/lib/ceph/osd/ceph-5 + CV_MODE=raw + DEVICE=/dev/sdc + cp --no-preserve=mode /etc/temp-ceph/ceph.conf /etc/ceph/ceph.conf + python3 -c ' import configparser config = configparser.ConfigParser() config.read('\''/etc/ceph/ceph.conf'\'') if not config.has_section('\''global'\''): config['\''global'\''] = {} if not config.has_option('\''global'\'','\''fsid'\''): config['\''global'\'']['\''fsid'\''] = '\''<redacted>'\'' with open('\''/etc/ceph/ceph.conf'\'', '\''w'\'') as configfile: config.write(configfile) ' + ceph -n client.admin auth get-or-create osd.5 mon 'allow profile osd' mgr 'allow profile osd' osd 'allow *' -k /etc/ceph/admin-keyring-store/keyring [osd.5] key = <redacted> + [[ raw == \l\v\m ]] ++ mktemp + OSD_LIST=/tmp/tmp.CekJVsr9gr + ceph-volume raw list /dev/sdc Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ self.main(self.argv) File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 166, in main self.list(args) File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 122, in list report = self.generate(args.device) File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 91, in generate info_device = [info for info in info_devices if info['NAME'] == dev][0] IndexError: list index out of range So it has failed executing `ceph-volume raw list /dev/sdc`. It looks like this code is new in 17.2.7. Is this a regression? What would be the simplest way to back out of it? Thanks, Matt -- Matthew Booth

1 3

Ceph dashboard reports CephNodeNetworkPacketErrors
by Dominique Ramaekers 07 Nov '23

07 Nov '23

Hi, I'm using Ceph on a 4-host cluster for a year now. I recently discovered the Ceph Dashboard :-) No I see that the Dashboard reports CephNodeNetworkPacketErrors >0.01% or >10 packets/s... Although all systems work great, I'm worried. 'ip -s link show eno5' results: 2: eno5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 7a:3b:79:9c:f6:d1 brd ff:ff:ff:ff:ff:ff permaddr 5c:ba:2c:08:b3:90 RX: bytes packets errors dropped missed mcast 734153938129 645770129 20160 0 0 342301 TX: bytes packets errors dropped carrier collsns 1085134190597 923843839 0 0 0 0 altname enp178s0f0 So in average 0,0003% of RX packet errors! All the four hosts uses the same 10Gb HP switch. The hosts themselves are HP Proliant G10 servers. I would expect 0% packet loss... Anyway. Should I be worried about data consistency? Or can Ceph handle this amount of packet errors? Greetings, Dominique.

2 2

Many pgs inactive after node failure
by Matthew Booth 06 Nov '23

06 Nov '23

I have a 3 node ceph cluster in my home lab. One of the pools spans 3 hdds, one on each node, and has size 2, min size 1. One of my nodes is currently down, and I have 160 pgs in 'unknown' state. The other 2 hosts are up and the cluster has quorum. Example `ceph health detail` output: pg 9.0 is stuck inactive for 25h, current state unknown, last acting [] I have 3 questions: Why would the pgs be in an unknown state? I would like to recover the cluster without recovering the failed node, primarily so that I know I can. Is that possible? The boot nvme of the host has failed, so I will most likely rebuild it. I'm running rook, and I will most likely delete the old node and create a new one with the same name. AFAIK, the OSDs are fine. When rook rediscovers the OSDs, will it add them back with data intact? If not, is there any way I can make it so it will? Thanks! -- Matthew Booth

3 3

Nautilus: Decommission an OSD Node
by Dave Hall 06 Nov '23

06 Nov '23

Hello., I've recently made the decision to gradually decommission my Nautilus cluster and migrate the hardware to a new Pacific or Quincy cluster. By gradually, I mean that as I expand the new cluster I will move (copy/erase) content from the old cluster to the new, making room to decommission more nodes and move them over. In order to do this I will, of course, need to remove OSD nodes by first emptying the OSDs on each node. I noticed that pgremapper (a version prior to October 2021) has a 'drain' subcommand that allows one to control which target OSDs would receive the PGs from the source OSD being drained. This seemed like a good idea: If one simply marks an OSD 'out', it's contents would be rebalanced to other OSDs on the same node that are still active, which seems like it would make a lot of unnecessary data movement and also make removing the next OSD take longer. So I went through the trouble of creating a 'really long' pgremapper drain command excluding the OSDs of two nodes as targets: # bin/pgremapper drain 16 --target-osds 00,01,02,03,04,05,06,07,24,25,16,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71 --allow-movement-across host --max-source-backfills 75 --concurrency 20 --verbose --yes However, when this is complete OSD 16 actually contains more PGs than before I started. It appears that the mapping generated by pgremapper also back-filled the OSD as it was draining it. So did I miss something here? What is the best way to proceed? I understand that it would be mayhem to mark 8 of 72 OSDs out and then turn backfill/rebalance/recover back on. But it seems like there should be a better way. Suggestions? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

OSD not starting
by Amudhan P 05 Nov '23

05 Nov '23

Hi, One of the server in Ceph cluster accidentally shutdown abruptly due to power failure. After restarting OSD's not coming up and in Ceph health check it shows osd down. When checking OSD status "osd.26 18865 unable to obtain rotating service keys; retrying" For every 30 seconds it's just putting a message and it's all the same in all OSD in the system. Nov 04 20:03:05 strg-node-03 bash[34287]: debug 2023-11-04T14:33:05.089+0000 7f1f5693c080 -1 osd.26 18865 unable to obtain rotating service keys; retrying Nov 04 20:03:35 strg-node-03 bash[34287]: debug 2023-11-04T14:33:35.090+0000 7f1f5693c080 -1 osd.26 18865 unable to obtain rotating service keys; retrying ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) on Debian 11 bullseye and cephadm based installation. Tried to search for errors and msg couldn't find anything useful. How do I fix this issue ? regards, Amudhan

2 2

RGW: Quincy 17.2.7 and rgw_crypt_default_encryption_key
by Jayanth Reddy 04 Nov '23

04 Nov '23

Hello Users, It is great to hear a note about RGW "S3 multipart uploads using Server-Side Encryption now replicate correctly in multi-site" in Quincy v17.2.7 release. But I see that users who are using [1] still have a dependency on the item tracked at [2]. I tested with Reef 18.2.0 as well and the PR [3] seems to be merged into reef but the configuration is not being effective. We're currently stuck at version 17.2.3 and in a situation where we can not upgrade to later because "rgw_crypt_default_encryption_key" is still WIP and also MPU with SSE (default key) is only fixed in 17.2.7 :( [1] https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo… [2] https://tracker.ceph.com/issues/61473 [3] https://github.com/ceph/ceph/pull/52796 Appreciate any help. Regards, Jayanth

1 0

resharding RocksDB after upgrade to Pacific breaks OSDs
by Denis Polom 03 Nov '23

03 Nov '23

Hi we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#r… <https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#r…>) to reshard RocksDB on our OSDs. Despite resharding reports operation as successful, OSD fails to start. # ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000 I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug. Thanks

7 8

ceph orch problem
by Dario Graña 03 Nov '23

03 Nov '23

Hi everyone! I've a ceph cluster running AlmaLinux9, podman and Ceph Quincy (17.6.0). Since yesterday I was having some problems but the last one is that the ceph orch command is hanging up. I have seen the logs but I didn't find anything relevant that could help to fix the problem. Podman shows the daemons running and if I stop one daemon it appears again after a few seconds. I also tried *ceph mgr fail* command, daemon order changes, but ceph orch still not working. ceph orch pause/resume also are not working. Disabling and enabling *cephadm* module didn't help. Any help to understand what's going on would be welcome. Thanks in advance. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 3

data corruption after rbd migration
by Nikola Ciprich 03 Nov '23

03 Nov '23

Dear ceph users and developers, we're struggling with strange issue which I think might be a bug causing snapshot data corruption while migrating RBD image we've tracked it to minimal set of steps to reproduce using VM with one 32G drive: rbd create --size 32768 sata/D2 virsh create xml_orig.xml rbd snap create ssd/D1@snap1 rbd export-diff ssd/D1@snap1 - | rbd import-diff - sata/D2 rbd export --export-format 1 --no-progress ssd/D1@snap1 - | xxh64sum 505dde3c49775773 rbd export --export-format 1 --no-progress sata/D2@snap1 - | xxh64sum 505dde3c49775773 # <- checksums match - OK virsh shutdown VM rbd migration prepare ssd/D1 sata/D1Z virsh create xml_new.xml rbd snap create sata/D1Z@snap2 rbd export-diff --from-snap snap1 sata/D1Z@snap2 - | rbd import-diff - sata/D2 rbd migration execute sata/D1Z rbd migration commit sata/D1Z rbd export --export-format 1 --no-progress sata/D1Z@snap2 - | xxh64sum 19892545c742c1de rbd export --export-format 1 --no-progress sata/D2@snap2 - | xxh64sum cc045975baf74ba8 # <- snapshosts differ OS is alma 9 based, kernel 5.15.105, CEPH 17.2.6, qemu-8.0.3 we tried disabling VM disk caches as well as discard, to no avail. my first question is, is it correct to assume creating snapshots while live migrating data is safe? if so, any ideas on where the problem could be? If I could provide more info, please let me know with regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

2 2

17.2.7 quincy
by Matthew Darwin 03 Nov '23

03 Nov '23

Hi all, I see17.2.7 quincy is published as debian-bullseye packages. So I tried it on a test cluster. I must say I was not expecting the big dashboard change in a patch release. Also all the "cluster utilization" numbers are all blank now (any way to fix it?), so the dashboard is much less usable now. Thoughts?

2 7

upgrade 17.2.6 to 17.2.7 , any issues?
by Dmitry Melekhov 03 Nov '23

03 Nov '23

Hello! I want to do upgrade from 17.2.6 to 17.2.7 , all 3 servers run alma 9. Could you share your experience? Any issues? Thank you!

2 2

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Emergency, I lost 4 monitors but all osd disk are safe
by Mohamed LAMDAOUAR 02 Nov '23

02 Nov '23

Hello, I have 7 machines on CEPH cluster, the service ceph runs on a docker container. Each machine has 4 hdd of data (available) and 2 nvme sssd (bricked) During a reboot, the ssd bricked on 4 machines, the data are available on the HDD disk but the nvme is bricked and the system is not available. is it possible to recover the data of the cluster (the data disk are all available)

7 14

"cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"
by Martin Conway 02 Nov '23

02 Nov '23

Hi I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses. I still can't get the reef version of cephadm to work properly. I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email. Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version" Traceback (most recent call last): File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9468, in <module> main() File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9456, in main r = ctx.func(ctx) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2108, in _infer_image ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2191, in infer_local_ceph_image container_info = get_container_info(ctx, daemon, daemon_name is not None) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in get_container_info matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in <listcomp> matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 217, in __getattr__ return super().__getattribute__(name) AttributeError: 'CephadmContext' object has no attribute 'fsid' I am running into other issues as well, but I think they may point back to this issue of "'CephadmContext' object has no attribute 'fsid'" Any help would be appreciated. Regards, Martin Conway IT and Digital Media Manager Research School of Physics Australian National University Canberra ACT 2601 +61 2 6125 1599 https://physics.anu.edu.au<https://physics.anu.edu.au/>

4 8

diskprediction_local module and trained models
by Can Özyurt 02 Nov '23

02 Nov '23

Hi everyone, We have recently noticed diskprediction_local module only works for a set of manufacturers. Hence we have the following questions: Are there any plans to support more manufacturers in the near future? Can we contribute to the process of training new models and how? Can the existing models be used with disks of other vendors (with hackish methods)? Is it that It just doesn't work or it would not be reliable? Basically how does the trained model and the vendor relation work? Say we have a disk for an existing model, what is the best way to test the trained model and the module itself? Thanks in advance

1 0

CephFS scrub causing MDS OOM-kill
by Denis Polom 02 Nov '23

02 Nov '23

Hi, I did setup CephFS forward scrub by executing cmd # ceph tell mds.cephfs:0 scrub start / recursive { "return_code": 0, "scrub_tag": "37a67f72-89a3-474e-8f8b-1e55cb979e14", "mode": "asynchronous" } But immediately after it started, memory usage on MDS that keeps rank 0 increased from 2GB to 32GB until MDS was killed by OOM killer. Why does it consume all the memory? Can it be adjusted somehow? Thank you

1 0

Ceph Leadership Team Meeting: 2023-11-1 Minutes
by Casey Bodley 01 Nov '23

01 Nov '23

quincy 17.2.7: released! * major 'dashboard v3' changes causing issues? https://github.com/ceph/ceph/pull/54250 did not merge for 17.2.7 * planning a retrospective to discuss what kind of changes should go in minor releases when members of the dashboard team are present reef 18.2.1: * most PRs already tested/merged * possibly start validation next week?

1 0

Moving devices to a different device class?
by Matt Larson 01 Nov '23

01 Nov '23

I am looking to create a new pool that would be backed by a particular set of drives that are larger nVME SSDs (Intel SSDPF2NV153TZ, 15TB drives). Particularly, I am wondering about what is the best way to move devices from one pool and to direct them to be used in a new pool to be created. In this case, the documentation suggests I could want to assign them to a new device-class and have a placement rule that targets that device-class in the new pool. Currently the Ceph cluster has two device classes 'hdd' and 'ssd', and the larger 15TB drives were automatically assigned to the 'ssd' device class that is in use by a different pool. The `ssd` device classes are used in a placement rule targeting that class. The documentation describes that I could set a device class for an OSD with a command like: `ceph osd crush set-device-class CLASS OSD_ID [OSD_ID ..]` Class names can be arbitrary strings like 'big_nvme". Before setting a new device class to an OSD that already has an assigned device class, should use `ceph osd crush rm-device-class ssd osd.XX`. Can I proceed to directly remove these OSDs from the current device class and assign to a new device class? Should they be moved one by one? What is the way to safely protect data from the existing pool that they are mapped to? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

4 5

negative list operation causing degradation in performance
by Vitaly Goot 01 Nov '23

01 Nov '23

In Ceph version 17.2.5, a notable performance drop is observed when executing list operations on non-existent objects, especially for large buckets undergoing high concurrency with frequent updates. For the context, initiating concurrent PUT/DELETE/LIST operations on 4 buckets, each containing ~10 million objects, initially yields satisfactory latencies. However, as updates continue to accumulate, the efficiency of the `negative list` operation diminishes notably. I suspect that over time, with many object insertions and deletions, the bucket index gets fragmented. While this affects all operations, negative lookups appear to be particularly hampered, possibly because they necessitate a comprehensive scan of the fragmented index. With the LIST arguments defined as $prefix=xxx $max-keys=2, are there any recommendations to optimize the Ceph configuration to enhance the performance of negative list operations?

1 0