Hey all,
I recently had a k8s node failure in my homelab, and even though I powered it off (and it's done for, so it won't get back up), it still shows up as watcher in rbd status.
```
root@node0:~# rbd status kubernetes/csi-vol-3e7af8ae-ceb6-4c94-8435-2f8dc29b313b
Watchers:
watcher=10.0.0.103:0/1520114202 client.1697844 cookie=140289402510784
watcher=10.0.0.103:0/39967552 client.1805496 cookie=140549449430704
root@node0:~# ceph osd blocklist ls
10.0.0.103:0/0 2023-04-15T13:15:39.061379+0200
listed 1 entries
```
Even though the node is down & I have blocked it multiple times for hours, it won't disappear. Meaning, ceph-csi-rbd claims the image is mounted already (manually binding works fine, and can cleanly unbind as well, but can't unbind from a node that doesn't exist anymore).
Is there any possibility to force kick an rbd client / watcher from ceph (e.g. switching the mgr / mon) or to see why this is not timing out?
I found some historical mails & issues (related to rook, which I don't use) regarding a param `osd_client_watch_timeout` but can't find how that relates to the RBD images.
Cheers,
Max.
Hi Team,
We are trying to integrate ceph with ovirt.
We have deployed ovirt 4.4.
We want to create a storage domain of POSIX compliant type for mounting a ceph based infrastructure in ovirt.
We have done SRV based resolution in our DNS server for ceph mon nodes but we are unable to create a storage domain using that.
We are able to manually mount the ceph-mon nodes using the following command on the deployment hosts:
sudo mount -t ceph :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 -o rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==
[root@deployment-host mnt]# df -kh
df: /run/user/0/gvfs: Transport endpoint is not connected
Filesystem Size Used Avail Use% Mounted on
[abcd:abcd:abcd::51]:6789,[abcd:abcd:abcd::52]:6789,[abcd:abcd:abcd::53]:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 19G 0 19G 0% /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9
Query:
1. Could anyone help us out with storage domain creation in ovirt for SRV resolved ceph-mon nodes.
Hi,
I am learning Ceph and I am having a hard time understanding PG and PG
calculus .
I know that a PG is a collection of objects, and that PG are replicated
over the hosts to respect the replication size, but...
In traditional storage, we use size in Gb, Tb and so on, we create a pool
from a bunch of disks or raid arrays of some size then we create volumes of
a certain size and use them. If the storage is full we add disks, then we
extend our pools/volumes.
The idea of size is simple to understand.
Ceph, although it supports the notion of pool size in Gb, Tb ...etc. Pools
are created using PGs, and now there is also the notion of % of data.
When I use pg calc from ceph or from redhat, the generated yml file
contains the % variable, but the commands file contains only the PGs, and
when you are configuring 15% and 18% have the same number of PGs
!!!!!!!!!!!!???
The pg calc encourages you to create a %data multiple of 100, in other
words, it assumes that you know all your pools from the start. What if you
won't consume all your raw disk space.
What happens when you need to add a new pool?
Also when you create several pools, and then execute ceph osd df tree, you
can see that all pools show the raw size as a free space, it is like all
pools share the same raw space regardless of their PG number.
If someone can put some light on this concept and how to manage it wisely,
because the documentation keeps saying that it's an important concept, that
you have to pay attention when choosing the number of PGs for a pool from
the start.
Regards.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…>
Virus-free.www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Hi folks,
Within a zonegroup, once a bucket is created, its metadata is sync-ed over to all zones. With bucket-level sync policy, however, its content may or may not be sync-ed over. To simplify the sync process, sometime I'd like to pick the bucket in a zone as the absolute truth and sync its content over to the bucket in another zone, which may have done some local deletions since the last time they were sync-ed. I don't want those local deletions to mess up with the planned sync. Is it possible to reset the bucket in this zone so it is in a "pristine" state and will receive everything from the source?
Thanks,Yixin
Hi,
I've noticed that when my lua script runs I get the following error on my
radosgw container. It looks like the lib64 directory is not included in the
path when looking for shared libraries.
Copying the content of lib64 into the lib directory solves the issue on the
running container.
Here are more details:
Apr 25 20:26:59 xxx-ceph-xxxx radosgw[60901]: req 2268223694354647302
0.000000000s Lua ERROR:
/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*share*/lua/5.3/socket.lua:12:
module 'socket.core' not found:
no field package.preload['socket.core']
no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*share*
/lua/5.3/socket/core.lua'
no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*lib*
/lua/5.3/socket/core.so'
no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*lib*
/lua/5.3/socket.so'
As mentioned the following command on the running radosgw container solves
the issue for the running container:
cp -ru /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib64/*
/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib/
Cheers,
Tom
Hi *,
I've set up grafana, prometheus and node-exporter on an adopted
cluster (currently running 16.2.10) and was trying to enable ssl for
grafana. As stated in the docs [1] there's a way to configure
individual certs and keys per host:
ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem
ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i $PWD/certificate.pem
So I did that, then ran 'ceph orch reconfig grafana' but I still get a
bad cert error message:
Apr 20 10:21:19 ceph01 conmon[3772491]: server.go:3160: http: TLS
handshake error from <IP>:46084: remote error: tls: bad certificate
It seems like the cephadm generated cert/key pair
(mgr/cephadm/grafana_key; mgr/cephadm/grafana_crt) supersedes the
per-host certs, and even after removing the generated cert/key (and
then reconfigure) cephadm regenerates a them and leaves me with the
same problem. Is this a known issue and what would be the fix? I
didn't find anything on tracker, but I might have missed it.
To confirm that my custom certs actually work I replaced the general
cert with my custom cert and the error doesn't appear, I can see the
grafana graphs in the dashboard. I could leave it like this, but if
grafana would failover it wouldn't work anymore, of course.
Any hints are greatly appreciated.
Thanks,
Eugen
[1]
https://docs.ceph.com/en/latest/cephadm/services/monitoring/#configuring-ss…
Actually, "bucket sync run" somehow made it worse since now the destination zone shows "bucket is caught up with source" from "bucket sync status" even though it clearly missed an object.
On Monday, April 24, 2023 at 02:37:46 p.m. EDT, Yixin Jin <yjin77(a)yahoo.ca> wrote:
An update:
After creating and enabling the bucket sync policy, I ran "bucket sync markers" and saw that each shard had the status of "init". The run "bucket sync run" in the end marked the status to be "incremental-sync", which seems to go through full-sync stage. However, the lone object in the source zone wasn't synced over to the destination zone.
I actually used gdb to walk through radosgw-admin to run "bucket sync run". It seems not to do anything for full-sync and it printed a log saying "finished iterating over all available prefixes:...", which actually broke off the do-while loop after the call to prefix_handler.revalidate_marker(&list_marker). This call returned false because it couldn't find rules from the sync pipe. I haven't drilled deeper to see why it didn't get rules, whatever it means. Nevertheless, the workaround with "bucket sync run" doesn't seem to work, at least not with Quincy.
Regards,Yixin
On Monday, April 24, 2023 at 12:37:24 p.m. EDT, Soumya Koduri <skoduri(a)redhat.com> wrote:
On 4/24/23 21:52, Yixin Jin wrote:
> Hello ceph gurus,
>
> We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1.
>
>
>
> Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync?
yes this is known_issue yet to be addressed with bucket level sync
policy ( - https://tracker.ceph.com/issues/57489 ). The interim
workaround to sync existing objects is to either
* create new objects (or)
* execute "bucket sync run"
after creating/enabling the bucket policy.
Please note that this issue is specific to only bucket policy but
doesn't exist for sync-policy set at zonegroup level.
Thanks,
Soumya
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hello ceph gurus,
We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1.
Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync?
I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. I also tried "bucket sync init" and "bucket sync run" via radosgw-admin. They don't seem to do what I expected. They simply mark z1 as not behind anymore but the single object still lives in z0 only.
I wonder how mature this sync policy feature is for production use.
Thanks,Yixin
We want to do the next urgent point release for pacific 16.2.13 ASAP.
The tip of the current pacific branch will be used as a base for this
release and we will build it later today.
Dev leads - if you have any outstanding PRs that must be included pls
merged them now.
Thx
YuriW
