- ceph-users - lists.ceph.io

Ceph Leadership Team Meeting Minutes Nov 15, 2023
by Ernesto Puerta 16 Nov '23

16 Nov '23

Hi Cephers, These are the topics discussed today: - 18.2.1 - Almost ready, packages built/signed - Plan to release on Monday - Last minute PR for Rook - Lab update to be finished by tomorrow - Finalize CDM APAC time - Review https://doodle.com/meeting/participate/id/aM9XGZ3a/vote?reauth=true - New time: 9.30 Pacific Time - Laura and Neha will sync up about changing the time - Squid dev freeze date - Proposal: end of January - Docs question: https://tracker.ceph.com/issues/11385: Can a member of the community just raise a PR attempting to standardize commands, without coordinating with a team? - ballpark date for pacific eol? docs still have the '2023-10-01' estimate - Discussion regarding EOL being at time of release or at some point shortly after to account for regressions - Conversation regarding clarity of messaging being important re: expectations for post-release fixes - concerns on stability of minor releases - tentatively this year - distro status update: still working to remove centos8/rhel8/ubuntu20 from main. https://github.com/ceph/ceph/pull/53901 stalled on container stuff in teuthology, and the need to rebuild containers with centos9 base - discuss the quincy/dashboard-v3 backports? was tabled from 11/1 [postponed to Nov 22] - Docs (Zac): CQ January 2024 https://pad.ceph.com/p/ceph_quarterly_2024_01 - Unittestability of dencoding: ask for review -- https://trello.com/c/R0h47dq2/870-unittestability-of-dencoding#comment-6553… - User + Dev meeting tomorrow - Need RGW representatives and people with EC profile knowledge Kind Regards, Ernesto

2 1

iSCSI GW trusted IPs
by Ramon Orrù 16 Nov '23

16 Nov '23

Hi, I’m configuring the iSCSI GW services on a quincy 17.2.3 cluster. I brought almost everything up and running (using cephadm), but I’m stuck in a configuration detail: if I check the gateway status in the Block -> iSCSI -> Overview section of the dashboard, they’re showing “Down” status, while the gateways are actually running. It makes me think the mgr is not able to talk with iSCSI APIs in order to collect info on the gateways, despite I correctly added my mgr hosts IPs to the trusted_ip_list parameter in my iscsi service definition yaml. While further checking the gateway logs I found some messages like: debug ::ffff:172.17.17.22 - - [15/Nov/2023 10:54:05] "GET /api/config?decrypt_passwords=True HTTP/1.1" 200 - debug ::ffff:172.17.17.22 - - [15/Nov/2023 10:54:05] "GET /api/_ping HTTP/1.1" 200 - debug ::ffff:172.17.17.22 - - [15/Nov/2023 10:54:05] "GET /api/gatewayinfo HTTP/1.1" 200 - Just after I reload the dashboard page. So I tried to add the 172.17.17.22 IP address to trusted_ip_list and it worked: iSCSI gateways status went green and Up on the dashboard. It sounds to me like it's some container private network address, but I can’t find any evidence of it when inspecting the containers cephadm spawned. My question is: how can I identify the IPs I need to make the iSCSI gateways properly reachable? I tried to add the whole 172.16.0.0/24 private class but no luck , the iscsi container starts but is not allowing 172.17.17.22 to access the APIs. Thanks in advance regards Ramon

3 2

migrate wal/db to block device
by Chris Dunlop 15 Nov '23

15 Nov '23

Hi, What's the correct way to migrate an OSD wal/db from a fast device to the (slow) block device? I have an osd with wal/db on a fast LV device and block on a slow LV device. I want to move the wal/db onto the block device so I can reconfigure the fast device before moving the wal/db back to the fast device. This link says to use "ceph-volume lvm migrate" (I'm on pacific, but the quincy and reef docs are the same): https://docs.ceph.com/en/pacific/ceph-volume/lvm/migrate/ I tried: $ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-volume lvm migrate --osd-id ${osdid} --osd-fsid ${osd_fsid} \ --from db wal --target ${block_vglv} $ systemctl stop ${osd_service} $ systemctl start ${osd_service} "cephadm ceph-volume lvm list" now shows only the (slow) block device whereas before the migrate it was showing both the block and db devices. However "lsof" shows the new osd process still has the original fast wal/db device open and "iostat" shows this device is still getting i/o. Also: $ ls -l /var/lib/ceph/${fsid}/osd.${osdid}/block* ...shows both the "block" and "block.db" symlinks to the original separate devices. And there are now no lv_tags on the original wal/db LV: $ lvs -o lv_tags ${original_db_vg_lv} Now I'm concerned there's device mismatch for this osd: "cephadm ceph-volume lvm list" believes there's no separate wal/db, but the osd is currently *using* the original separate wal/db. I guess if the server were to restart this osd would be in all sorts of trouble. What's going on there, and what can be done to fix it? Is it a matter of recreating the tags on the original db device? (But then what happens to whatever did get migrated to the block device - e.g. is that space lost?) Or is it a matter of using ceph-bluestore-tool to do a bluefs-bdev-migrate, e.g. something like: $ cephadm unit --fsid ${fsid} --name osd.${osdid} stop $ osddir=/var/lib/ceph/osd/ceph-${osdid} $ cephadm shell --fsid ${fsid} --name osd.${osdid} -- \ ceph-bluestore-tool --path ${osddir} --devs-source ${osddir}/block.db \ --dev-target ${osddir}/block bluefs-bdev-migrate $ rm /var/lib/ceph/${fsid}/osd.${osdid}/block.db $ systemctl stop ${osd_service} $ systemctl start ${osd_service} Or... something else? And how *should* moving the wal/db be done? Cheers, Chris

3 6

Large size differences between pgs
by Miroslav Svoboda 15 Nov '23

15 Nov '23

Hi,is it possible decrease large size differences between pgs? I have 5PB cluster and differences between smalest and bigest pgs are somewhere about 25GB.thanks,Svoboda Miroslav

1 0

per-rbd snapshot limitation
by Wesley Dillingham 15 Nov '23

15 Nov '23

looking into how to limit snapshots at the ceph level for RBD snapshots. Ideally ceph would enforce an arbitrary number of snapshots allowable per rbd. Reading the man page for rbd command I see this option: https://docs.ceph.com/en/quincy/man/8/rbd/#cmdoption-rbd-limit --limit Specifies the limit for the number of snapshots permitted. Seems perfect. But on attempting to use it as such I get an error: admin@rbdtest:~$ rbd create testpool/test3 --size=100M --limit=3 rbd: unrecognised option '--limit=3' Where am I going wrong here? Is there another way to enforce a limit of snapshots for RBD? Thanks. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

3 5

Join us for the User + Dev Monthly Meetup - November 16!
by Laura Flores 15 Nov '23

15 Nov '23

Hi Ceph users and developers, You are invited to join us at the User + Dev meeting this week Thursday, November 16th at 10:00 AM EST! See below for more meeting details. The focus topic, "Operational Reliability and Flexibility in Ceph Upgrades", will be presented by Christian Theune. His presentation will highlight some issues encountered in a long running cluster when upgrading to stable releases, including migration between EC profiles, challenges related to RGW zone replication, and low-level bugs that need more attention from developers. The last part of the meeting will be dedicated to open discussion. Feel free to add questions for the speakers or additional topics under the "Open Discussion" section on the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have an idea for a focus topic you'd like to present at a future meeting, you are welcome to submit it to this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Any Ceph user or developer is eligible to submit! Thanks, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time conversions: UTC: Thursday, November 16, 15:00 UTC Mountain View, CA, US: Thursday, November 16, 7:00 PST Phoenix, AZ, US: Thursday, November 16, 8:00 MST Denver, CO, US: Thursday, November 16, 8:00 MST Huntsville, AL, US: Thursday, November 16, 9:00 CST Raleigh, NC, US: Thursday, November 16, 10:00 EST London, England: Thursday, November 16, 15:00 GMT Paris, France: Thursday, November 16, 16:00 CET Helsinki, Finland: Thursday, November 16, 17:00 EET Tel Aviv, Israel: Thursday, November 16, 17:00 IST Pune, India: Thursday, November 16, 20:30 IST Brisbane, Australia: Friday, November 17, 1:00 AEST Singapore, Asia: Thursday, November 16, 23:00 +08 Auckland, New Zealand: Friday, November 17, 4:00 NZDT -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

rasize= in ceph.conf some section?
by Pat Riehecky 15 Nov '23

15 Nov '23

Hello, I'm trying to make it easy to distribute the expected config settings for my ceph volumes for other admin groups. Is there a place I can set rasize in the ceph.conf where the client would pick it up? The NFS world these days has nfsmount.conf that I've grown very fond of. I'm a bit concerned that there will be drift in rasize and whatnot that causes performance issues. While it doesn't seem much, having two places to look for mount information is less ideal....

1 0

ceph -s very slow in my rdma eviroment
by WeiGuo Ren 15 Nov '23

15 Nov '23

Today I run some ceph -s in my rdma enviroment. but very slow. After perf ceph -s to FlameGraph. I find almost all the time is spent on compact zone. Has anyone encountered it? my enviroment . rpm -qa | grep ibverb libibverbs-41mlnx1-OFED.4.1.0.1.1.41102.x86_64 libibverbs-devel-41mlnx1-OFED.4.1.0.1.1.41102.x86_64 ceph -v 14.2.5

1 0

planning upgrade from pacific to quincy
by Simon Oosthoek 15 Nov '23

15 Nov '23

Hi All (apologies if you get this twice, I suspect mails from my @science.ru.nl account get dropped by most receiving mail servers, due to the strict DMARC policies in place) after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we are now planning to upgrade our cluster which is running Pacific (at least on the mons/mdss/osds, the gateways are by accident running quincy already). The installation is via packages from ceph.com, unless it's quincy from ubuntu. ceph versions: "mon": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "mgr": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "osd": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 252, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12 }, "mds": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 2 }, "rgw": {"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 }, "overall": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 260, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 } The OS on the mons and mdss are still ubuntu 18.04, the osds are a mix of ubuntu 18 and ubuntu 20. The gateways are ubuntu 22.04, which is why these are already on quincy. The plan is to move to quincy and eventually cephadm/containered ceph, since that is apparently "the way to go", though I have my doubts. The steps we think are the right order are: - reinstall the mons with ubuntu 22.04 + quincy - reinstall the osds (same) - reinstall the mdss (same) Once this is up and running, we want to investigate and migrate to cephadm orchestration. Alternative appear to be: move to orchestration first and then upgrade ceph to quincy (possibly skipping the ubuntu upgrade?) Another alternative could be to upgrade to quincy on ubuntu 18.04 using packages, but I haven't investigated the availability of quincy packages for ubuntu 18.04 (which is out of free (LTS) support by canonical) Cheers /Simon

1 0

planning upgrade from pacific to quincy
by Simon Oosthoek 15 Nov '23

15 Nov '23

Hi All after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we are now planning to upgrade our cluster which is running Pacific (at least on the mons/mdss/osds, the gateways are by accident running quincy already). The installation is via packages from ceph.com, unless it's quincy from ubuntu. ceph versions: "mon": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "mgr": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "osd": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 252, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12 }, "mds": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 2 }, "rgw": {"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 }, "overall": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 260, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 } The OS on the mons and mdss are still ubuntu 18.04, the osds are a mix of ubuntu 18 and ubuntu 20. The gateways are ubuntu 22.04, which is why these are already on quincy. The plan is to move to quincy and eventually cephadm/containered ceph, since that is apparently "the way to go", though I have my doubts. The steps we think are the right order are: - reinstall the mons with ubuntu 22.04 + quincy - reinstall the osds (same) - reinstall the mdss (same) Once this is up and running, we want to investigate and migrate to cephadm orchestration. Alternative appear to be: move to orchestration first and then upgrade ceph to quincy (possibly skipping the ubuntu upgrade?) Another alternative could be to upgrade to quincy on ubuntu 18.04 using packages, but I haven't investigated the availability of quincy packages for ubuntu 18.04 (which is out of free (LTS) support by canonical) Cheers /Simon

1 0