- ceph-users - lists.ceph.io

MGR Memory Leak in Restful
by Chris Palmer 08 Sep '23

08 Sep '23

I first posted this on 17 April but did not get any response (although IIRC a number of other posts referred to it). Seeing as MGR OOM is being discussed at the moment I am re-posting. These clusters are not containerized. Is this being tracked/fixed or not? Thanks, Chris ------------------------------- We've hit a memory leak in the Manager Restful interface, in versions 17.2.5 & 17.2.6. On our main production cluster the active MGR grew to about 60G until the oom_reaper killed it, causing a successful failover and restart of the failed one. We can then see that the problem is recurring, actually on all 3 of our clusters. We've traced this to when we enabled full Ceph monitoring by Zabbix last week. The leak is about 20GB per day, and seems to be proportional to the number of PGs. For some time we just had the default settings, and no memory leak, but had not got around to finding why many of the Zabbix items were showing as Access Denied. We traced this to the MGR's MON CAPS which were "mon 'profile mgr'". The MON logs showed recurring: log_channel(audit) log [DBG] : from='mgr.284576436 192.168.xxx.xxx:0/2356365' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: access denied Changing the MGR CAPS to "mon 'allow *'" and restarting the MGR immediately allowed that to work, and all the follow-on REST calls worked. log_channel(audit) log [DBG] : from='mgr.283590200 192.168.xxx.xxx:0/1779' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: dispatch However it has also caused the memory leak to start. We've reverted the CAPS and are back to how we were. Two questions: 1) No matter what the REST consumer is doing, the MGR should not accumulate memory, especially as we can see that the REST TCP connections have wrapped up. Is there anything more we can do to diagnose this? 2) Setting "allow *" worked, but is there are better setting just to allow the "pg dump" call (in addition to profile mgr)? Thanks, Chris

2 1

Re: Rocksdb compaction and OSD timeout
by xiaowenhao111 07 Sep '23

07 Sep '23

1 0

failure domain and rack awareness
by Reza Bakhshayeshi 07 Sep '23

07 Sep '23

Hello, What is the best strategy regarding failure domain and rack awareness when there are only 2 physical racks and we need 3 replicas of data? In this scenario what is your point of view if we create 4 artificial racks at least to be able to manage deliberate node maintenance in a more efficient way? Regards, Reza

1 0

Permissions of the .snap directory do not inherit ACLs in 17.2.6
by MARTEL Arnaud 07 Sep '23

07 Sep '23

Hi, I'm facing the same situation as described in bug #57084 (https://tracker.ceph.com/issues/57084) since I upgraded from 16.2.13 to 17.2.6 for example: root@faiserver:~# getfacl /mnt/ceph/default/ # file: mnt/ceph/default/ # owner: 99 # group: nogroup # flags: -s- user::rwx user:s-sac-acquisition:rwx group::rwx group:acquisition:r-x group:SAC_R:r-x mask::rwx other::--- default:user::rwx default:user:s-sac-acquisition:rwx default:group::rwx default:group:acquisition:r-x default:group:SAC_R:r-x default:mask::rwx default:other::--- root@faiserver:~# getfacl /mnt/ceph/default/.snap # file: mnt/ceph/default/.snap # owner: 99 # group: nogroup # flags: -s- user::rwx group::rwx other::r-x </pre> Before creating a new bug report, could you tell me if someone has the same problem with 17.2.6 ?? Kind regards, Arnaud

2 3

Is it possible (or meaningful) to revive old OSDs?
by ceph-mail＠rikdvk.mailer.me 07 Sep '23

07 Sep '23

Hello, I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since. I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start. Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes". I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster? My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard? [1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-o… [2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-… Many thanks!

5 5

Re: Debian/bullseye build for reef
by Josh Durgin 07 Sep '23

07 Sep '23

We weren't targeting bullseye once we discovered the compiler version problem, the focus shifted to bookworm. If anyone would like to help maintaining debian builds, or looking into these issues, it would be welcome: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129 https://tracker.ceph.com/issues/61845 On Mon, Aug 21, 2023 at 7:50 AM Matthew Darwin <bugs(a)mdarwin.ca> wrote: > Thanks for the link to the issue. Any reason it wasn't added to the > release notes (for bullseye). > > I am also waiting for this to be available to start testing. > On 2023-08-21 10:25, Josh Durgin wrote: > > There was difficulty building on bullseye due to the older version of GCC > available: https://tracker.ceph.com/issues/61845 > > On Mon, Aug 21, 2023 at 3:01 AM Chris Palmer <chris.palmer(a)idnet.com> <chris.palmer(a)idnet.com> wrote: > > > I'd like to try reef, but we are on debian 11 (bullseye). > In the ceph repos, there is debian-quincy/bullseye and > debian-quincy/focal, but under reef there is only focal & jammy. > > Is there a reason why there is no reef/bullseye build? I had thought > that the blocker only affected debian-bookworm builds. > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 2

Contionuous spurious repairs without cause?
by Christian Theune 06 Sep '23

06 Sep '23

Hi, this is a bit older cluster (Nautilus, bluestore only). We’ve noticed that the cluster is almost continuously repairing PGs. However, they all finish successfully with “0 fixed”. We do not see the trigger why Ceph decides to repair the PGs and it’s happening for a lot of PGs, not any specific individual one. Deep-scrubs are generally running, but currently a bit late as we had some recoveries in the last week. Logs look regular aside from the number of repairs. Here’s the last weeks from the perspective of a single PG. There’s one repair, but the same thing seems to happen for all PGs. 2023-08-06 16:08:17.870 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-06 16:08:18.270 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-07 21:52:22.299 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-07 21:52:22.711 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-09 00:33:42.587 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-09 00:33:43.049 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-10 09:36:00.590 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-10 09:36:28.811 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-11 12:59:14.219 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-11 12:59:14.567 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-12 13:52:44.073 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-12 13:52:44.483 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-14 01:51:04.774 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-14 01:51:33.113 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-15 05:18:16.093 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-15 05:18:16.520 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-16 09:47:38.520 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-16 09:47:38.930 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-17 19:25:45.352 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-17 19:25:45.775 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-19 05:40:43.663 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-19 05:40:44.073 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-20 12:06:54.343 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-20 12:06:54.809 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-21 19:23:10.801 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-21 19:23:39.936 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-23 03:43:21.391 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-23 03:43:21.844 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-24 04:21:17.004 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-24 04:21:47.972 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-25 06:55:13.588 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-25 06:55:14.087 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-26 09:26:01.174 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-26 09:26:01.561 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-27 11:18:10.828 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-27 11:18:11.264 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-28 19:05:42.104 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-28 19:05:42.693 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-30 07:03:10.327 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-30 07:03:10.805 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-31 14:43:23.849 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-31 14:43:50.723 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-09-01 20:53:42.749 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-01 20:53:43.389 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-02 22:57:49.542 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-02 22:57:50.065 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-04 03:16:14.754 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-04 03:16:15.295 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-05 14:50:36.064 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 repair starts 2023-09-05 14:51:04.407 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 repair ok, 0 fixed Googling didn’t help, unfortunately and the bug tracker doesn’t appear to have any relevant issue either. Any ideas? Liebe Grüße, Christian Theune -- Christian Theune · ct(a)flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

3 4

Join Us for the Relaunch of the Ceph User + Developer Monthly Meeting!
by Laura Flores 06 Sep '23

06 Sep '23

Hi Ceph users and developers, We’re happy to announce a relaunch of the Ceph User + Developer Monthly Meeting – a virtual platform that has long been in our community for encouraging discussion and collaboration between users and developers. With this relaunch, we aim to recenter the platform around user-facing topics that go beyond immediate bug reports, such as long-term improvements and knowledge sharing. Users and developers are encouraged to submit focus topics to this Google form <https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v…> [1] in preparation for each meeting. Read more about what we're looking for in focus topics here: https://ceph.io/en/news/blog/2023/user-dev-meeting-relaunch/ Join us on *September 21st, 10:00am EST *at this link <https://meet.jit.si/ceph-user-dev-monthly> [2] to witness the relaunch! - Laura Flores 1. User + Dev Google form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… 2. Meeting link: https://meet.jit.si/ceph-user-dev-monthly -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Questions about 'public network' and 'cluster nertwork'?
by Louis Koo 06 Sep '23

06 Sep '23

if the public network and cluster network use the same ip, why still need to send heartbeats to hb_front_server and hb _back_server at same time?

1 0

insufficient space ( 10 extents) on vgs lvm detected locked
by absankar89＠gmail.com 06 Sep '23

06 Sep '23

ceph orch device ls - output ( insufficient space ( 10 extents) on vgs lvm detected locked ) Quincy version , Is this just warning or any action should be taken.

2 1