On 11/05/2022 23:21, Joost Nieuwenhuijse wrote:
> After a reboot the OSD turned out to be corrupt. Not sure if
> ceph-volume lvm new-db caused the problem, or failed because of
> another problem.
I just ran into the same issue trying to add a db to an existing OSD.
Apparently this is a known bug: https://tracker.ceph.com/issues/55260
It's already fixed master, but the backports are all still pending ...
Regards
Christian
Hi, experts,
we have a product env build with ceph version 16.2.11 pacific, and using CephFS.
Also enable multi active mds(more than 10), but we usually see load unbalance on our client request with these mds.
see below picture. the top 1 mds has 32.2k client request. and the last one only 331.
this always lead our cluster into very bad situation. say many MDS report slow requests…
...
7 MDSs report slow requests
1 MDSs behind on trimming
…
So our question is how to set those mdss load balance? Could any one please help to shed some light here?
Thanks a ton!
Thanks,
xz
Hello,
We started to use the Ceph bucket notification events with subscription to an HTTP endpoint.
We encountered an issue when the receiver endpoint was changed. Which means the events from Ceph weren't consumed. We deleted the bucket notifications and the topic, and created a new topic with the new endpoint and new bucket notifications.
(We are using the REST api to create bucket notifications and topics. We also used the CLI commands, but there we found out that deleting a topic doesn't delete the notifications that are subscribed to it. Ceph version is Pacific.)
From that moment we didn't receive any more notification events to our new endpoint.
We tried many times to create new topics and new bucket notifications, but we don't receive anymore events to our endpoint.
We suspect that the notification queues don't get fully cleaned and they stay in some broken state.
We have been able to reproduce this locally and the only solution was to wipe all the containers and recreate them. The problem is that this issue is on a staging environment where we cannot destroy everything.
We are looking for a solution or a command to clean the notification queues, to be able to start anew.
We also are looking for a way to know programatically if the notifications broke and have a way to automatically recover as such a flaw is critical for our application.
Thanks for your time!
Daniel Yordanov
Libcephfs's 'init' call hangs when passed arguments that once worked
normally, but later refer to a cluster that's either broken, is on its
way out of service, has too few mons, etc. At least the python
libcephfs wrapper hangs on init.
Of course mount and session timeouts work, but is there a way to error
out a failed init call and not just hang the client?
Thanks!
Today we discussed:
- Delegating more privileges for internal hardware to allow on-call
folks to fix issues.
- Maybe using CephFS for the teuthology VM /home directory (it became
full on Friday night)
- Preparation for Open Source Day: we are seeking "low-hanging-fruit"
tickets for new developers to try fixing.
- Reef is released! Time for blog posts. We are gathering options from PTLs.
- Ceph organization Github plan migration from the "bronze legacy
plan" to the FOSS "free" plan. There is some uncertainty about
surprise drawbacks, Ernesto is continuing his investigation.
- Case is updating contributors to generate accurate credits for the
new reef release: https://github.com/ceph/ceph/pull/52868
--
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
I am in the process of expanding our cluster capacity by ~50% and have
noticed some unexpected behavior during the backfill and recovery process
that I'd like to understand and see if there is a better configuration that
will yield a faster and smoother backfill.
Pool Information:
OSDs: 243 spinning HDDs
PGs: 1024 (yes, this is low for our number of disks)
I inherited the cluster and it has the following settings which seem to
have been done in an attempt to get the cluster to recover quickly:
osd_max_backfills: 6 (default is 1)
osd_recovery_sleep_hdd: 0.0 (default is 0.1)
osd_recovery_max_active_hdd: 9
When watching the PGs recover I am noticing a few things:
- All PGs seem to be backfilling at the same time which seems to be in
violation of osd_max_backfills. I understand that there should be 6 readers
and 6 writers at a time, but I'm seeing a given OSD participate in more
than 6 PG backfills. Is an OSD only considered as backfilling if it is not
present in both the UP and ACTING groups (e.g. it will have it's data
altered)?
- Some PGs are recovering at a much slower rate than others (some as little
as kilobytes per second) despite the disks being all of a similar speed. Is
there some way to dig into why that may be?
- In general, the recovery is happening very slowly (between 1 and 5
objects per second per PG). Is it possible the settings above are too
aggressive and causing performance degradation due to disk thrashing?
- Currently, all misplaced PGs are backfilling, if I were to change some of
the settings above (specifically `osd_max_backfills`) would that
essentially pause backfilling PGs or will those backfills have to end and
then start over when it is done waiting?
- Given that all PGs are backfilling simultaneously there is no way to
prioritize one PG over another (we have some disks with very high usage
that we're trying to reduce). Would reducing those max backfills allow for
proper prioritization of PGs with force-backfill?
- We have had some OSDs restart during the process and their misplaced
object count is now zero but they are incrementing their recovering objects
bytes. Is that expected and is there a way to estimate when that will
complete?
Thanks for the help!
-Jonathan
Hi all,
I have an rbd image that `rbd disk-usage` shows it has 31GB usage but in
the filesystem `du` shows its usage is 40KB.
Does anyone know the reason for this difference?
Best Regards,
Mahnoosh
Hi,
I have a ceph stucked at `ceph --verbose stats fs fsname`. And in the
monitor log, I can found something like `audit [DBG] from='client.431973 -'
entity='client.admin' cmd=[{"prefix": "fs status", "fs": "fsname",
"target": ["mon-mgr", ""]}]: dispatch`.
What happened and what should I do?
--
ZhangBao
+6585021702
We have a FreeBSD 12.3 guest machine that works well on an RBD volume
until it is live migrated to another node (on Proxmox). After
migration, the processes almost all go into D state (waiting for this
disk) and they don't exist from it (ie they don't "get" the disk the
requested.
I'm not sure what it causing this, so I'm asking here if anyone has come
across such a problem?
