Hi Everyone,
I've been fighting with a ceph cluster that we have recently
physically relocated and lost 2 OSDs during the ensuing power down and
relocation. After powering everything back on we have
3 incomplete
3 remapped+incomplete
And indeed we have 2 OSDs that died along the way.
The reason I'm contacting the list is that I'm surprised that these
PGs are incomplete. We're running Erasure coding with K=4, M=2 which
in my understanding we should be able to lose 2 OSDs without an issue.
Am I mis-understanding this or does m=2 mean you can lose m-1 OSDs?
Also, these two OSDs happened to be in the same server (#3 of 8 total servers).
This is an older cluster running Nautilus 14.2.9.
Any thoughts?
Thanks
-Dave
I first posted this on 17 April but did not get any response (although
IIRC a number of other posts referred to it).
Seeing as MGR OOM is being discussed at the moment I am re-posting.
These clusters are not containerized.
Is this being tracked/fixed or not?
Thanks, Chris
-------------------------------
We've hit a memory leak in the Manager Restful interface, in versions
17.2.5 & 17.2.6. On our main production cluster the active MGR grew to
about 60G until the oom_reaper killed it, causing a successful failover
and restart of the failed one. We can then see that the problem is
recurring, actually on all 3 of our clusters.
We've traced this to when we enabled full Ceph monitoring by Zabbix last
week. The leak is about 20GB per day, and seems to be proportional to
the number of PGs. For some time we just had the default settings, and
no memory leak, but had not got around to finding why many of the Zabbix
items were showing as Access Denied. We traced this to the MGR's MON
CAPS which were "mon 'profile mgr'".
The MON logs showed recurring:
log_channel(audit) log [DBG] : from='mgr.284576436
192.168.xxx.xxx:0/2356365' entity='mgr.host1' cmd=[{"format": "json",
"prefix": "pg dump"}]: access denied
Changing the MGR CAPS to "mon 'allow *'" and restarting the MGR
immediately allowed that to work, and all the follow-on REST calls worked.
log_channel(audit) log [DBG] : from='mgr.283590200
192.168.xxx.xxx:0/1779' entity='mgr.host1' cmd=[{"format": "json",
"prefix": "pg dump"}]: dispatch
However it has also caused the memory leak to start.
We've reverted the CAPS and are back to how we were.
Two questions:
1) No matter what the REST consumer is doing, the MGR should not
accumulate memory, especially as we can see that the REST TCP
connections have wrapped up. Is there anything more we can do to
diagnose this?
2) Setting "allow *" worked, but is there are better setting just to
allow the "pg dump" call (in addition to profile mgr)?
Thanks, Chris
Hello,
What is the best strategy regarding failure domain and rack awareness when
there are only 2 physical racks and we need 3 replicas of data?
In this scenario what is your point of view if we create 4 artificial racks
at least to be able to manage deliberate node maintenance in a more
efficient way?
Regards,
Reza
Hi,
I'm facing the same situation as described in bug #57084 (https://tracker.ceph.com/issues/57084) since I upgraded from 16.2.13 to 17.2.6
for example:
root@faiserver:~# getfacl /mnt/ceph/default/
# file: mnt/ceph/default/
# owner: 99
# group: nogroup
# flags: -s-
user::rwx
user:s-sac-acquisition:rwx
group::rwx
group:acquisition:r-x
group:SAC_R:r-x
mask::rwx
other::---
default:user::rwx
default:user:s-sac-acquisition:rwx
default:group::rwx
default:group:acquisition:r-x
default:group:SAC_R:r-x
default:mask::rwx
default:other::---
root@faiserver:~# getfacl /mnt/ceph/default/.snap
# file: mnt/ceph/default/.snap
# owner: 99
# group: nogroup
# flags: -s-
user::rwx
group::rwx
other::r-x
</pre>
Before creating a new bug report, could you tell me if someone has the same problem with 17.2.6 ??
Kind regards,
Arnaud
Hello,
I have a ten node cluster with about 150 OSDs. One node went down a while back, several months. The OSDs on the node have been marked as down and out since.
I am now in the position to return the node to the cluster, with all the OS and OSD disks. When I boot up the now working node, the OSDs do not start.
Essentially , it seems to complain with "fail[ing]to load OSD map for [various epoch]s, got 0 bytes".
I'm guessing the OSDs on disk maps are so old, they can't get back into the cluster?
My questions are whether it's possible or worth it to try to squeeze these OSDs back in or to just replace them. And if I should just replace them, what's the best way? Manually remove [1] and recreate? Replace [2]? Purge in dashboard?
[1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#removing-o…
[2] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-…
Many thanks!
We weren't targeting bullseye once we discovered the compiler version
problem, the focus shifted to bookworm. If anyone would like to help
maintaining debian builds, or looking into these issues, it would be
welcome:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129https://tracker.ceph.com/issues/61845
On Mon, Aug 21, 2023 at 7:50 AM Matthew Darwin <bugs(a)mdarwin.ca> wrote:
> Thanks for the link to the issue. Any reason it wasn't added to the
> release notes (for bullseye).
>
> I am also waiting for this to be available to start testing.
> On 2023-08-21 10:25, Josh Durgin wrote:
>
> There was difficulty building on bullseye due to the older version of GCC
> available: https://tracker.ceph.com/issues/61845
>
> On Mon, Aug 21, 2023 at 3:01 AM Chris Palmer <chris.palmer(a)idnet.com> <chris.palmer(a)idnet.com> wrote:
>
>
> I'd like to try reef, but we are on debian 11 (bullseye).
> In the ceph repos, there is debian-quincy/bullseye and
> debian-quincy/focal, but under reef there is only focal & jammy.
>
> Is there a reason why there is no reef/bullseye build? I had thought
> that the blocker only affected debian-bookworm builds.
>
> Thanks, Chris
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>