I have hell of the question: how to make HEALTH_ERR status for a cluster
without consequences?
I'm working on CI tests and I need to check if our reaction to
HEALTH_ERR is good. For this I need to take an empty cluster with an
empty pool and do something. Preferably quick and reversible.
For HEALTH_WARN the best thing I found is to change pool size to 1, it
raises "1 pool(s) have no replicas configured" warning almost instantly
and it can be reverted very quickly for empty pool.
But HEALTH_ERR is a bit more tricky. Any ideas?
Hi,
Is there anybody tried consul as a load balancer?
Any experience?
Thank you
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Dear Community,
Since Nautilus, we have 2 mechanisms for notifying 3rd parties on changes
in buckets and objects: "bucket notifications" [1] and "pubsub" [2].
In "bucket notifications" (="push mode") the events are sent from the RGW
to an external entity (kafka, rabbitmq etc.), while in "pubsub" (="pull
mode") the events are synched with a special zone, where they are stored
and could be later fetched by an external app.
From communications that I've seen so far, users preferred to use "bucket
notifications" over "pubsub". Since supporting both modes has maintenance
overhead, I was considering deprecating "pubsub".
However, before doing that I would like to see what the community has to
say!
So, if you are currently using pubsub, or plan to use it, as "pull mode"
fits your usecase better than "push mode" please chime in.
Yuval
[1] https://docs.ceph.com/en/latest/radosgw/notifications/
[2] https://docs.ceph.com/en/latest/radosgw/pubsub-module/
Hi all,
During rejoin an MDS can sometimes go OOM if the openfiles table is too large.
The workaround has been described by ceph devs as "rados rm -p
cephfs_metadata mds0_openfiles.0".
On our cluster we have several such objects for rank 0:
mds0_openfiles.0 exists with size: 199978
mds0_openfiles.1 exists with size: 153650
mds0_openfiles.2 exists with size: 40987
mds0_openfiles.3 exists with size: 7746
mds0_openfiles.4 exists with size: 413
If we suffer such an OOM, do we need to rm *all* of those objects or
only the `.0` object?
Best Regards,
Dan
Hi,
I found 6-700 stale instances with the reshard stale instances list command.
Is there a way to clean it up (or actually should I clean it up)?
The stale instance rm doesn't work in multisite.
Thank you
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hi all,
I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a single new OSD daemon with a db device on SSD. But I can’t find any documentation about this.
I have tried:
1. Using web dashboard. This requires at least one filter to proceed (type, vendor, model or size). But I just want to select the block device manually.
2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based.
3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db device in this way.
4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’ does.
5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I can’t figure out what is that.
Any help is appreciated. Thanks.
Hello everyone,
We just installed a Ceph cluster version luminous (12.2.11) on servers
working with Debian buster (10.8)
using ceph-deploy and we are trying to upgrade it to mimic but can't
find a way to do it.
We tried ceph-deploy install --release mimic mon1 mon2 mon3 (after
having modified /etc/apt/sources.list.d/ceph.list)
but this does nothing because the packets are said to be up to date.
Could someone help us, please ?
Best regards
Hi,
Today while debugging something we had a few questions that might lead
to improving the cephfs forward scrub docs:
https://docs.ceph.com/en/latest/cephfs/scrub/
tldr:
1. Should we document which sorts of issues that the forward scrub is
able to fix?
2. Can we make it more visible (in docs) that scrubbing is not
supported with multi-mds?
3. Isn't the new `ceph -s` scrub task status misleading with multi-mds?
Details here:
1) We found a CephFS directory with a number of zero sized files:
# ls -l
...
-rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58
upload_fc501199e3e7abe6b574101cf34aeefb.png
-rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23
upload_fce4f55348185fefa0abdd8d11095ba8.gif
-rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54
upload_fd95b8358851f0dac22fb775046a6163.png
...
The user claims that those files were non-zero sized last week. The
sequence of zero sized files includes *all* files written between Nov
2 and 9.
The user claims that his client was running out of memory, but this is
now fixed. So I suspect that his ceph client (kernel
3.10.0-1127.19.1.el7.x86_64) was not behaving well.
Anyway, I noticed that even though the dentries list 0 bytes, the
underlying rados objects have data, and the data looks good. E.g.
# rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx
200212e68b5.00000000
# file 200212e68b5.00000000
200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA,
non-interlaced
So I managed to recover the files doing something like this (using an
input file mapping inode to filename) [see PS 0].
But I'm wondering if a forward scrub is able to fix this sort of
problem directly?
Should we document which sorts of issues that the forward scrub is able to fix?
I anyway tried to scrub it, which led to:
# ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx
recursive repair
Scrub is not currently supported for multiple active MDS. Please
reduce max_mds to 1 and then scrub.
So ...
2) Shouldn't we update the doc to mention loud and clear that scrub is
not currently supported for multiple active MDS?
3) I was somehow surprised by this, because I had thought that the new
`ceph -s` multi-mds scrub status implied that multi-mds scrubbing was
now working:
task status:
scrub status:
mds.x: idle
mds.y: idle
mds.z: idle
Is it worth reporting this task status for cephfs if we can't even scrub them?
Thanks!!
Dan
[0]
mkdir -p recovered
while read -r a b; do
for i in {0..9}
do
echo "rados stat --cluster=flax --pool=cephfs_data
--namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get
--cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x"
$a).0000000$i $(printf "%x" $a).0000000$i
done
echo cat $(printf "%x" $a).* ">" $(printf "%x" $a)
echo mv $(printf "%x" $a) recovered/$b
done < inones_fnames.txt
