- ceph-users - lists.ceph.io

RBD tuning for virtualization (all flash)
by by morphin 24 Apr '21

24 Apr '21

Hello. We're running 1000vm on 28 node with 6 ssd (no seperate db device) and these vms are Mostly win10. 2 lvm osd Per 4tb device total 288osd and one RBD pool with 8192 PG. Replication 3. Ceph version : nautilus 14.2.16 I'm looking for all flash RBD tuning. This is good test env and tomorrow gonna be prod. In vms I see 1800mb/s read and 900mb/s write (qemu = write back) With good tuning I believe this cluster will go further. Do you have any special suggestion? Have a nice weekend.

1 0

Pacific, one of monitor service doesnt response.
by Cem Zafer 24 Apr '21

24 Apr '21

Hi, I have no problem with my cluster before but when upgraded from 15.2.11 to 16.2.0 one of my ceph monitor service does not work. I have looked at the service log but cant figure it out. Can someone have any idea what could be wrong with that monitor service? Here is the service attached log. Regards root@ceph100:~# ceph orch ps NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID alertmanager.ceph100 ceph100 running (102m) 7m ago 3d - 0.20.0 0881eb8f169f b62ac302b319 crash.ceph100 ceph100 running (102m) 7m ago 3d - 16.2.0 24ecd6d5f14c e786dc48b429 crash.ceph101 ceph101 running (99m) 5m ago 3d - 16.2.0 24ecd6d5f14c a2cefd673a5f crash.ceph102 ceph102 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c f9f8a337ba43 grafana.ceph100 ceph100 running (102m) 7m ago 3d - 6.7.4 80728b29ad3f 60381a67b6a2 mgr.ceph100.abnyjw ceph100 running (102m) 7m ago 3d *:8443 *:9283 16.2.1 c757e4a3636b 67f98b3ad44d mgr.ceph102.qqcrik ceph102 running (101m) 7m ago 8h *:8443 *:9283 16.2.1 c757e4a3636b 00a6eeb2a9eb mon.ceph100 ceph100 running (102m) 7m ago 3d - 16.2.0 24ecd6d5f14c 6bce0a137567 mon.ceph101 ceph101 stopped 5m ago 3d - <unknown> <unknown> <unknown> mon.ceph102 ceph102 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c 9ca0e1c7ecd9 node-exporter.ceph100 ceph100 running (102m) 7m ago 3d - 0.18.1 e5a616e4b9cf b8ae7aa0fae1 node-exporter.ceph101 ceph101 running (99m) 5m ago 3d - 0.18.1 e5a616e4b9cf 945cdd081979 node-exporter.ceph102 ceph102 running (101m) 7m ago 3d - 0.18.1 e5a616e4b9cf d9a967358fbe osd.0 ceph100 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c 8f74ba97c27e osd.1 ceph100 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c fe0eed6a2cf4 osd.2 ceph101 running (99m) 5m ago 3d - 16.2.0 24ecd6d5f14c 1fd53036a139 osd.3 ceph101 running (99m) 5m ago 3d - 16.2.0 24ecd6d5f14c 329496f3a03e osd.4 ceph102 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c cb16b76f0fc4 osd.5 ceph102 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c c5d4e56812af prometheus.ceph100 ceph100 running (102m) 7m ago 3d - 2.18.1 de242295e225 b3f1c51d7136

2 1

Different ceph versions on nodes in cluster
by mabi 24 Apr '21

24 Apr '21

Hello, I was wondering if it is ok to have different Ceph version numbers among the nodes in a Ceph cluster? I am asking because I started setting up a new Octopus cluster with cephadm bootstrap which installed my initial node with version 15.2.10 and added a new mon node today and that new node got installed with 15.2.11... What is the best practice here? Best regards, Mabi

3 4

How to get rbd block device IOPS and BW performance showing?
by huxiaoyu＠horebdata.cn 24 Apr '21

24 Apr '21

Dear Ceph folks, I have a Luminous ceph cluster serving rbd block devices for an openstack cluster. My team leader would like me to show the performance view of these rbd block devices, including IOPS and BW, to get to know which rbd consumes most. I heard that Nautilous can do this , but how to get those statistics for Luminous version? Any help would be highly appreciated, thanks in advance, Samuel huxiaoyu(a)horebdata.cn

2 1

wrong socket path with ceph daemonperf
by Fabrice Bacchella 24 Apr '21

24 Apr '21

I doing a custom, manual installation of ceph. I changed the cluster name from the default one to 'ngceph'. I changed the installation path to /data/ceph. A lot of thing correctly resolves the settings. But one command fails: CLUSTER=/data/ceph/conf/ngceph.conf CLUSTER=ngceph ceph daemonperf client.radosgw.fa43 list ... Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ceph_daemon.py", line 73, in admin_socket b'{"prefix": "get_command_descriptions"}') File "/usr/lib/python3.6/site-packages/ceph_daemon.py", line 49, in do_sockio sock.connect(path) FileNotFoundError: [Errno 2] No such file or directory ... And indeed, after a few check, I found the culprit: CLUSTER=/data/ceph/conf/ngceph.conf CLUSTER=ngceph strace -f -e trace=file ceph daemonperf client.radosgw.fa43 list ... [pid 3431] execve("/bin/ceph-conf", ["ceph-conf", "--name", "client.radosgw.fa43", "--show-config-value", "admin_socket"], 0x5651c6e1cb30 /* 31 vars */) = 0 So I tried this command: CLUSTER=/data/ceph/conf/ngceph.conf CLUSTER=ngceph ceph-conf --name client.radosgw.fa43 --show-config-value admin_socket /var/run/ceph/ceph-client.radosgw.fa43.asok But the real path for the socket is: ls -l /var/run/ceph/ngceph-client.radosgw.fa43.2521.94787303273792.asok srwxr-xr-x 1 ceph ceph 0 Apr 22 21:47 /var/run/ceph/ngceph-client.radosgw.fa43.2521.94787303273792.asok It does not use the cluster name, and I wonder what is this 2521.94787303273792 coming from. Would it fails even with a standard cluster name ? The version I'm using is a rpm build from ceph: $ rpm -qf /bin/ceph-conf ceph-common-16.2.1-0.el8.x86_64

2 4

ceph osd replace SD card
by For 99 24 Apr '21

24 Apr '21

Hi every body. I have ceph cluster. One of OSD host os on sd card that have 6 osd hdd is crashed, so I need to change sd card. how can I reconnect 6 osd hard disks to new os without destroy the data? Thanks in advance.

3 2

Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data
by 特木勒 23 Apr '21

23 Apr '21

Thank you so much~ I will follow your guide to repair index and let-sync the bucket. 😬 Will let you know the result after I do this. Thanks for your help again~ Amit Ghadge <amitg.b14(a)gmail.com>于2021年4月23日周五下午10:19写道： > that my steps, it may be helpfull, > > if buckets objects not sync and bucket sync not work then fix index in > master and then run bucket sync > master:radosgw-admin bucket check -b bucket1 --fix --check-objects > secondary:radosgw-admin bucket sync run --source-zone us-west > --bucket=bucket1 > > > On Fri, Apr 23, 2021 at 7:07 PM 特木勒 <twl007(a)gmail.com> wrote: > >> I have not tried to fix bucket index. >> >> For error list I have checked and it looks good. >> >> The strange thing is only the objects that created after multiple site >> will be synced. And after sync recovered from networking issues, secondary >> cluster could pull loss data during networking downtime. So look like it’s >> working, just I do not know why only old objects will not be synced.🙁 >> >> Thanks for your suggestion and I will give it a try. >> >> Have a good day~ >> >> Amit Ghadge <amitg.b14(a)gmail.com>于2021年4月23日周五下午9:00写道： >> >>> Maybe you can try to fix bucket Index from source zone or you have to >>> check is there any error in error list. >>> >>> On Fri, 23 Apr 2021 at 6:20 PM, 特木勒 <twl007(a)gmail.com> wrote: >>> >>>> Hi Istvan: >>>> >>>> We just upgraded whole cluster to 15.2.10 and the multiple site still >>>> cannot sync whole objects to secondary cluster. 🙁 >>>> >>>> Do you have any suggestions on this? And I open another issues in ceph >>>> tracker site: >>>> https://tracker.ceph.com/issues/50474 >>>> >>>> Hope someone could go to check this issue. >>>> >>>> Thanks >>>> >>>> 特木勒 <twl007(a)gmail.com>于2021年3月22日周一下午9:08写道： >>>> >>>> > Thank you~ >>>> > >>>> > I will try to upgrade cluster too. Seem like this is the only way for >>>> now. >>>> > 😭 >>>> > >>>> > I will let you know once I complete testing. :) >>>> > >>>> > Have a good day >>>> > >>>> > Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>于2021年3月22日周一下午3:38写道： >>>> > >>>> >> Yeah, doesn't work. Last week they fixed my problem ticket which >>>> caused >>>> >> the crashes, and due to the crashes stopped the replication I'll >>>> give a try >>>> >> this week again after the update if the daemon doesn't crash, maybe >>>> it will >>>> >> work, because if crash hasn't happened, the data was synced. Fingers >>>> >> crossed ;) Don't give up 😄 >>>> >> ------------------------------ >>>> >> *From:* 特木勒 <twl007(a)gmail.com> >>>> >> *Sent:* Monday, March 22, 2021 1:38 PM >>>> >> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >>>> >> *Cc:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> >>>> >> >>>> >> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple >>>> Site >>>> >> does not sync olds data >>>> >> >>>> >> Hi Istvan: >>>> >> >>>> >> Do you have any update on directional sync? >>>> >> >>>> >> I am trying to upgrade cluster to 15.2.10 to see if the problem is >>>> >> solved. :( >>>> >> >>>> >> Thanks >>>> >> >>>> >> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年3月1日周一 >>>> 上午10:01写道： >>>> >> >>>> >> So-so. I had some interruption so it failed on one site, but the >>>> other is >>>> >> kind of working. This is the first time when I saw data caught up in >>>> the >>>> >> radosgw-admin data sync status on 1 side. >>>> >> >>>> >> Today will finish the other problematic site, I’ll let you know the >>>> >> result is it working or not. >>>> >> >>>> >> >>>> >> >>>> >> Istvan Szabo >>>> >> Senior Infrastructure Engineer >>>> >> --------------------------------------------------- >>>> >> Agoda Services Co., Ltd. >>>> >> e: istvan.szabo(a)agoda.com >>>> >> --------------------------------------------------- >>>> >> >>>> >> >>>> >> >>>> >> *From:* 特木勒 <twl007(a)gmail.com> >>>> >> *Sent:* Sunday, February 28, 2021 1:34 PM >>>> >> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >>>> >> *Cc:* ceph-users(a)ceph.io >>>> >> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple >>>> Site >>>> >> does not sync olds data >>>> >> >>>> >> >>>> >> >>>> >> Email received from outside the company. If in doubt don't click >>>> links >>>> >> nor open attachments! >>>> >> ------------------------------ >>> >>> >>>> >> >>>> >> Hi Istvan: >>>> >> >>>> >> >>>> >> >>>> >> Thanks for your reply. >>>> >> >>>> >> >>>> >> >>>> >> Does directional sync solve the problem? I tried to run >>>> `radosgw-admin >>>> >> sync init`, bit it still did not work. :( >>>> >> >>>> >> >>>> >> >>>> >> Thanks >>>> >> >>>> >> >>>> >> >>>> >> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年2月26日周五 >>>> 上午7:47写道： >>>> >> >>>> >> Same for me, 15.2.8 also. >>>> >> I’m trying directional sync now, looks like symmetrical has issue. >>>> >> >>>> >> Istvan Szabo >>>> >> Senior Infrastructure Engineer >>>> >> --------------------------------------------------- >>>> >> Agoda Services Co., Ltd. >>>> >> e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> >>>> >> --------------------------------------------------- >>>> >> >>>> >> On 2021. Feb 26., at 1:03, 特木勒 <twl007(a)gmail.com> wrote: >>>> >> >>>> >> Email received from outside the company. If in doubt don't click >>>> links >>>> >> nor open attachments! >>>> >> ________________________________ >>>> >> >>>> >> Hi all: >>>> >> >>>> >> ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) >>>> >> >>>> >> I have a strange question, I just create a multiple site for Ceph >>>> cluster. >>>> >> But I notice the old data of source cluster is not synced. Only new >>>> data >>>> >> will be synced into second zone cluster. >>>> >> >>>> >> Is there anything I need to do to enable full sync for bucket or >>>> this is a >>>> >> bug? >>>> >> >>>> >> Thanks >>>> >> _______________________________________________ >>>> >> ceph-users mailing list -- ceph-users(a)ceph.io >>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >> >>>> >> ________________________________ >>>> >> This message is confidential and is for the sole use of the intended >>>> >> recipient(s). It may also be privileged or otherwise protected by >>>> copyright >>>> >> or other legal rules. If you have received it by mistake please let >>>> us know >>>> >> by reply email and delete it from your system. It is prohibited to >>>> copy >>>> >> this message or disclose its content to anyone. Any confidentiality >>>> or >>>> >> privilege is not waived or lost by any mistaken delivery or >>>> unauthorized >>>> >> disclosure of the message. All messages sent to and from Agoda may be >>>> >> monitored to ensure compliance with company policies, to protect the >>>> >> company's interests and to remove potential malware. Electronic >>>> messages >>>> >> may be intercepted, amended, lost or deleted, or contain viruses. >>>> >> >>>> >> >>>> >> Internal >>>> >> >>>> > >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>

1 0

Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data
by 特木勒 23 Apr '21

23 Apr '21

I have not tried to fix bucket index. For error list I have checked and it looks good. The strange thing is only the objects that created after multiple site will be synced. And after sync recovered from networking issues, secondary cluster could pull loss data during networking downtime. So look like it’s working, just I do not know why only old objects will not be synced.🙁 Thanks for your suggestion and I will give it a try. Have a good day~ Amit Ghadge <amitg.b14(a)gmail.com>于2021年4月23日周五下午9:00写道： > Maybe you can try to fix bucket Index from source zone or you have to > check is there any error in error list. > > On Fri, 23 Apr 2021 at 6:20 PM, 特木勒 <twl007(a)gmail.com> wrote: > >> Hi Istvan: >> >> We just upgraded whole cluster to 15.2.10 and the multiple site still >> cannot sync whole objects to secondary cluster. 🙁 >> >> Do you have any suggestions on this? And I open another issues in ceph >> tracker site: >> https://tracker.ceph.com/issues/50474 >> >> Hope someone could go to check this issue. >> >> Thanks >> >> 特木勒 <twl007(a)gmail.com>于2021年3月22日周一下午9:08写道： >> >> > Thank you~ >> > >> > I will try to upgrade cluster too. Seem like this is the only way for >> now. >> > 😭 >> > >> > I will let you know once I complete testing. :) >> > >> > Have a good day >> > >> > Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>于2021年3月22日周一下午3:38写道： >> > >> >> Yeah, doesn't work. Last week they fixed my problem ticket which caused >> >> the crashes, and due to the crashes stopped the replication I'll give >> a try >> >> this week again after the update if the daemon doesn't crash, maybe it >> will >> >> work, because if crash hasn't happened, the data was synced. Fingers >> >> crossed ;) Don't give up 😄 >> >> ------------------------------ >> >> *From:* 特木勒 <twl007(a)gmail.com> >> >> *Sent:* Monday, March 22, 2021 1:38 PM >> >> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >> >> *Cc:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> >> >> >> >> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >> >> does not sync olds data >> >> >> >> Hi Istvan: >> >> >> >> Do you have any update on directional sync? >> >> >> >> I am trying to upgrade cluster to 15.2.10 to see if the problem is >> >> solved. :( >> >> >> >> Thanks >> >> >> >> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年3月1日周一上午10:01写道： >> >> >> >> So-so. I had some interruption so it failed on one site, but the other >> is >> >> kind of working. This is the first time when I saw data caught up in >> the >> >> radosgw-admin data sync status on 1 side. >> >> >> >> Today will finish the other problematic site, I’ll let you know the >> >> result is it working or not. >> >> >> >> >> >> >> >> Istvan Szabo >> >> Senior Infrastructure Engineer >> >> --------------------------------------------------- >> >> Agoda Services Co., Ltd. >> >> e: istvan.szabo(a)agoda.com >> >> --------------------------------------------------- >> >> >> >> >> >> >> >> *From:* 特木勒 <twl007(a)gmail.com> >> >> *Sent:* Sunday, February 28, 2021 1:34 PM >> >> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >> >> *Cc:* ceph-users(a)ceph.io >> >> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >> >> does not sync olds data >> >> >> >> >> >> >> >> Email received from outside the company. If in doubt don't click links >> >> nor open attachments! >> >> ------------------------------ > > >> >> >> >> Hi Istvan: >> >> >> >> >> >> >> >> Thanks for your reply. >> >> >> >> >> >> >> >> Does directional sync solve the problem? I tried to run `radosgw-admin >> >> sync init`, bit it still did not work. :( >> >> >> >> >> >> >> >> Thanks >> >> >> >> >> >> >> >> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年2月26日周五上午7:47写道： >> >> >> >> Same for me, 15.2.8 also. >> >> I’m trying directional sync now, looks like symmetrical has issue. >> >> >> >> Istvan Szabo >> >> Senior Infrastructure Engineer >> >> --------------------------------------------------- >> >> Agoda Services Co., Ltd. >> >> e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> >> >> --------------------------------------------------- >> >> >> >> On 2021. Feb 26., at 1:03, 特木勒 <twl007(a)gmail.com> wrote: >> >> >> >> Email received from outside the company. If in doubt don't click links >> >> nor open attachments! >> >> ________________________________ >> >> >> >> Hi all: >> >> >> >> ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) >> >> >> >> I have a strange question, I just create a multiple site for Ceph >> cluster. >> >> But I notice the old data of source cluster is not synced. Only new >> data >> >> will be synced into second zone cluster. >> >> >> >> Is there anything I need to do to enable full sync for bucket or this >> is a >> >> bug? >> >> >> >> Thanks >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> ________________________________ >> >> This message is confidential and is for the sole use of the intended >> >> recipient(s). It may also be privileged or otherwise protected by >> copyright >> >> or other legal rules. If you have received it by mistake please let us >> know >> >> by reply email and delete it from your system. It is prohibited to copy >> >> this message or disclose its content to anyone. Any confidentiality or >> >> privilege is not waived or lost by any mistaken delivery or >> unauthorized >> >> disclosure of the message. All messages sent to and from Agoda may be >> >> monitored to ensure compliance with company policies, to protect the >> >> company's interests and to remove potential malware. Electronic >> messages >> >> may be intercepted, amended, lost or deleted, or contain viruses. >> >> >> >> >> >> Internal >> >> >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

1 0

certificates between msg and radosgw
by Fabrice Bacchella 23 Apr '21

23 Apr '21

I've just activated certificates on my radosgw, and the manager is unable to talk to it. On the manager logs, I see: 2021-04-23T15:05:03.917+0200 7f4109309700 0 [dashboard ERROR urllib3.connection] Certificate did not match expected hostname: XXX Certificate: {...)} 2021-04-23T15:05:03.918+0200 7f4109309700 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/lib/python3.6/site-packages/urllib3/connection.py", line 364, in connect _match_hostname(cert, self.assert_hostname or server_hostname) File "/lib/python3.6/site-packages/urllib3/connection.py", line 374, in _match_hostname match_hostname(cert, asserted_hostname) File "/lib64/python3.6/ssl.py", line 287, in match_hostname % (hostname, ', '.join(map(repr, dnsnames)))) ssl.CertificateError: hostname 'SOMEIP' doesn't match either of 'XXX' So the manager is communicating to the gateway using the IP. So I configured the names in ceph.conf: [client.radosgw.ID] rgw_data = /data/ceph/data/radosgw/$cluster.ID log_file = /data/ceph/logs/$cluster-radosgw.ID.log rgw_frontends = "beast ssl_endpoint=0.0.0.0:443 ssl_certificate=/data/ceph/conf/ceph.crt ssl_private_key=/data/ceph/conf/ceph.key" rgw_dns_name = XXX But it's seems that the line rgw_dns_name is not used by the manager. Where does it resolve the name to use for the connection ?

1 0

Re: rbd snap create now working and just hangs forever
by Ilya Dryomov 23 Apr '21

23 Apr '21

On Fri, Apr 23, 2021 at 1:12 PM Boris Behrens <bb(a)kervyn.de> wrote: > > > > Am Fr., 23. Apr. 2021 um 13:00 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> On Fri, Apr 23, 2021 at 12:46 PM Boris Behrens <bb(a)kervyn.de> wrote: >> > >> > >> > >> > Am Fr., 23. Apr. 2021 um 12:16 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> >> >> On Fri, Apr 23, 2021 at 12:03 PM Boris Behrens <bb(a)kervyn.de> wrote: >> >> > >> >> > >> >> > >> >> > Am Fr., 23. Apr. 2021 um 11:52 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> >> >> >> >> >> >> >> This snippet confirms my suspicion. Unfortunately without a verbose >> >> >> log from that VM from three days ago (i.e. when it got into this state) >> >> >> it's hard to tell what exactly went wrong. >> >> >> >> >> >> The problem is that the VM doesn't consider itself to be the rightful >> >> >> owner of the lock and so when "rbd snap create" requests the lock from >> >> >> it in order to make a snapshot, the VM just ignores the request because >> >> >> even though it owns the lock, its record appears to be of sync. >> >> >> >> >> >> I'd suggest to kick it by restarting osd36. If the VM is active, it >> >> >> should reacquire the lock and hopefully update its internal record as >> >> >> expected. If "rbd snap create" still hangs after that, it would mean >> >> >> that we have a reproducer and can gather logs on the VM side. >> >> >> >> >> >> What version of qemu/librbd and ceph is in use (both on the VM side and >> >> >> on the side you are running "rbd snap create"? >> >> >> >> >> > I just stopped the OSD, waited some seconds and started it again. >> >> > I still can't create snapshots. >> >> > >> >> > Ceph version is 14.2.18 accross the board >> >> > qemu is 4.1.0-1 >> >> > as we use krbd, the kernel version is 5.2.9-arch1-1-ARCH >> >> > >> >> > How can I gather more logs to debug it? >> >> >> >> Are you saying that this image is mapped and the lock is held by the >> >> kernel client? It doesn't look that way from the logs you shared. >> > >> > We use krbd instead of librbd (at least this is what I think I know), but qemu is doing the kvm/rbd stuff. >> >> I'm going to assume that by "qemu is doing the kvm/rbd stuff", you >> mean that you are using the librbd driver inside qemu and that this >> image is opened by qemu (i.e. that driver). If you don't know what >> access method is being used, debugging this might be challenging ;) >> >> Let's start with the same output: "rbd lock ls", "rbd status" and "rbd >> snap create --debug-ms=1 --debug-rbd=20". It should be different after >> osd36 was restarted. > > Here is the new one: https://pastebin.com/6qTsJK6W > Ah ok, this CPU node still got the old thing and uses librbd to work with rbd instead of krbd. Sorry, I forgot that simply restarting the OSD doesn't trigger the code path that I'm hoping would cause librbd inside the VM to update its state. I took a look at the code and I think there are a couple of ways to do it (listed in the order of preference): - cut the network between the VM and the cluster for more than 30 seconds; it should be done externally so that to the VM it looks like a long network blip - stop the VM process for more than 30 seconds $ PID=<pid of qemu-system-x86_64 process> $ kill -STOP $PID && sleep 40 && kill -CONT $PID - stop the osd36 process for more than 30 seconds with "nodown" flag set $ ceph osd set nodown $ PID=<pid of osd36 process> $ kill -STOP $PID && sleep 40 && kill -CONT $PID $ ceph osd unset nodown Thanks, Ilya

1 0