- ceph-users - lists.ceph.io

[Quincy] cannot configure dashboard to listen on all ports
by wodel youchi 04 Mar '24

04 Mar '24

Hi, ceph dashboard fails to listen on all IPs. log_channel(cluster) log [ERR] : Unhandled exception from module 'dashboard' while running on mgr.controllera: OSError("No socket could be created -- (('0.0.0.0', 8443): [Errno -2] Name or service not known) -- (('::', 8443, 0, 0): ceph version 17.2.7 quincy (stable) Regards.

1 0

ceph-crash NOT reporting crashes due to wrong permissions on /var/lib/ceph/crash/posted (Debian / Ubuntu packages)
by Christian Rohmann 04 Mar '24

04 Mar '24

Hey ceph-users, I just noticed issues with ceph-crash using the Debian /Ubuntu packages (package: ceph-base): While the /var/lib/ceph/crash/posted folder is created by the package install, it's not properly chowned to ceph:ceph by the postinst script. This might also affect RPM based installs somehow, but I did not look into that. I opened a bug report with all the details and two ideas to fix this: https://tracker.ceph.com/issues/64548 The wrong ownership causes ceph-crash to NOT work at all. I myself missed quite a few crash reports. All of them were just sitting around on the machines, but were reported right after I did chown ceph:ceph /var/lib/ceph/crash/posted systemctl restart ceph-crash.service You might want to check if you might be affected as well. Failing to post crashes to the local cluster results in them not being reported back via telemetry. Regards Christian

3 3

Ceph orch doesn't execute commands and doesn't report correct status of daemons
by wodel youchi 03 Mar '24

03 Mar '24

Hi, I have finished the conversion from ceph-ansible to cephadm yesterday. Everything seemed to be working until this morning, I wanted to redeploy rgw service to specify the network to be used. So I deleted the rgw services with ceph orch rm, then I prepared a yml file with the new conf. I applied the file and the new rgw service was started but it was launched with an external image, so I wanted to redeploy using my local image so I did a redeploy ... and then nothing happened, I get the rescheduled message but nothing happened, then I restarted one of the controllers, the orchestrator doesn't seem to be aware that some service have restarted??? PS : I don't fully master the cephadm command line and use. Regards.

2 3

[Quincy] NFS ingress mode haproxy-protocol not recognized
by wodel youchi 03 Mar '24

03 Mar '24

Hi; I tried to create an NFS cluster using this command : [root@controllera ceph]# ceph nfs cluster create mynfs "3 controllera controllerb controllerc" --ingress --virtual_ip 20.1.0.201 --ingress-mode haproxy-protocol Invalid command: haproxy-protocol not in default|keepalive-only And I got this error : Invalid command haproxy-protocol I am using Quincy : ceph version 17.2.7 (...) quincy (stable) Is it not supported yet? Regards.

2 1

Question about erasure coding on cephfs
by Erich Weiler 03 Mar '24

03 Mar '24

Hi Y'all, We have a new ceph cluster online that looks like this: md-01 : monitor, manager, mds md-02 : monitor, manager, mds md-03 : monitor, manager store-01 : twenty 30TB NVMe OSDs store-02 : twenty 30TB NVMe OSDs The cephfs storage is using erasure coding at 4:2. The crush domain is set to "osd". (I know that's not optimal but let me get to that in a minute) We have a current regular single NFS server (nfs-01) with the same storage as the OSD servers above (twenty 30TB NVME disks). We want to wipe the NFS server and integrate it into the above ceph cluster as "store-03". When we do that, we would then have three OSD servers. We would then switch the crush domain to "host". My question is this: Given that we have 4:2 erasure coding, would the data rebalance evenly across the three OSD servers after we add store-03 such that if a single OSD server went down, the other two would be enough to keep the system online? Like, with 4:2 erasure coding, would 2 shards go on store-01, then 2 shards on store-02, and then 2 shards on store-03? Is that how I understand it? Thanks for any insight! -erich

3 2

Re: has anyone enabled bdev_enable_discard?
by jsterr＠deckplosion.de 02 Mar '24

02 Mar '24

Is there any update on this? Did someone test the option and has performance values before and after? Is there any good documentation regarding this option?

7 7

Re: Scrub stuck and 'pg has invalid (post-split) stat'
by Eugen Block 01 Mar '24

01 Mar '24

Please don't drop the list from your response. The first question coming to mind is, why do you have a cache-tier if all your pools are on nvme decices anyway? I don't see any benefit here. Did you try the suggested workaround and disable the cache-tier? Zitat von Cedric <yipikai7(a)gmail.com>: > Thanks Eugen, see attached infos. > > Some more details: > > - commands that actually hangs: ceph balancer status ; rbd -p vms ls ; > rados -p vms_cache cache-flush-evict-all > - all scrub running on vms_caches pgs are stall / start in a loop > without actually doing anything > - all io are 0 both from ceph status or iostat on nodes > > On Tue, Feb 20, 2024 at 10:00 AM Eugen Block <eblock(a)nde.ag> wrote: >> >> Hi, >> >> some more details would be helpful, for example what's the pool size >> of the cache pool? Did you issue a PG split before or during the >> upgrade? This thread [1] deals with the same problem, the described >> workaround was to set hit_set_count to 0 and disable the cache layer >> until that is resolved. Afterwards you could enable the cache layer >> again. But keep in mind that the code for cache tier is entirely >> removed in Reef (IIRC). >> >> Regards, >> Eugen >> >> [1] >> https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-addin… >> >> Zitat von Cedric <yipikai7(a)gmail.com>: >> >> > Hello, >> > >> > Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we >> > encounter an issue with a cache pool becoming completely stuck, >> > relevant messages below: >> > >> > pg xx.x has invalid (post-split) stats; must scrub before tier agent >> > can activate >> > >> > In OSD logs, scrubs are starting in a loop without succeeding for all >> > pg of this pool. >> > >> > What we already tried without luck so far: >> > >> > - shutdown / restart OSD >> > - rebalance pg between OSD >> > - raise the memory on OSD >> > - repeer PG >> > >> > Any idea what is causing this? any help will be greatly appreciated >> > >> > Thanks >> > >> > Cédric >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 5

What's up with 16.2.15?
by Zakhar Kirpichenko 01 Mar '24

01 Mar '24

Hi, We randomly got several Pacific package updates to 16.2.15 available for Ubuntu 20.04. As far as I can see, 16.2.15 hasn't been released and there's been no release announcement. The updates seem to be no longer available. What's going on with 16.2.15? /Z

2 1

Renaming an OSD node
by Deep Dish 01 Mar '24

01 Mar '24

Hello. We have a requirement to change the hostname on some of our OSD nodes. All of our nodes are Ubuntu 22.04 based and have been deployed using 17.2.7 Orchestrator. 1. Is there a procedure to rename the existing node, without rebuilding and have it detected by Ceph Orchestrator? If not, 2. To minimize impact on cluster (rebuilding OSDs / balancing, etc) Is it possible to REINTRODUCE existing OSDs into the cluster a the newly rebuilt node? Is there a ceph orch process to scan local node OSDs, detect and create OSD daemons? Thank you.

2 1

Slow RGW multisite sync due to "304 Not Modified" responses on primary zone
by praveenkumargpk17＠gmail.com 01 Mar '24

01 Mar '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our Grafana instance and out of 1000 requests / second, 200 requests are getting "200 OK" and 800 requests are getting "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks , Praveen

1 0