does the RBD client block write when the Watcher times out?
Hello. I'm currently verifying the behavior of RBD on failure. I'm wondering about the consistency of RBD images after network failures. As a result of my investigation, I found that RBD sets a Watcher to RBD image if a client mounts this volume to prevent multiple mounts. In addition, I found that if the client is isolated from the network for a long time, the Watcher is released. However, the client still mounts this image. In this situation, if another client can also mount this image and the image is writable from both clients, data corruption occurs. Could you tell me whether this is a realistic scenario? I tested the following case on which the watcher was released by hand and detected data corruption. 1. Release the Watcher on the node (A) that mounts the RBD using the `ceph osd blocklist add` command 2. Another node (B) mounts the RBD volume. 3. Unblock using the `ceph osd blocklist rm` command 4. Write from node (B) (write successfully) 5. Write from node (A) (can be written successfully from the application's point of view. In fact, the write fails) 6. Write content at node (A) is lost. In this case, I released the watcher by hand to emulate the timeout due to network failure. It's because I couldn't emulate real network failure in this test. I considered using exclusive lock and restricting access to those from a single node. However, we gave up on that as blocking writes entirely would make snapshots non-functional. The version of Ceph we are using is v17.2.6. Best regards, Yuma.
I found that if the client is isolated from the network for a long time, the Watcher is released. However, the client still mounts this image. In this situation, if another client can also mount this image and the image is writable from both clients, data corruption occurs. Could you tell me whether this is a realistic scenario? I've had a similar scenario which resulted in a pretty nasty corruption. RBD image was mapped on a client which got suspended, then changes were made by another client, forgetting that the image was still mapped. It is realistic and cannot be avoided easily as it's difficult to differentiate between a network downtime and sudden host shutdown (power loss/system crash/etc...).
I considered using exclusive lock and restricting access to those from a single node. However, we gave up on that as blocking writes entirely would make snapshots non-functional. Exclusive locks are a option, but i've encountered a pretty nasty bug with them under high load on krbd. I have not reported this problem yet, but if my memory serves me right, it happened during network instability with lots of I/O pending.
-- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/
On Thu, May 23, 2024 at 4:48 AM Yuma Ogami <yuma.ogami.cybozu@gmail.com> wrote:
Hello.
I'm currently verifying the behavior of RBD on failure. I'm wondering about the consistency of RBD images after network failures. As a result of my investigation, I found that RBD sets a Watcher to RBD image if a client mounts this volume to prevent multiple mounts. In
Hi Yuma, The watcher is created to watch for updates (technically, to listen to notifications) on the RBD image, not to prevent multiple mounts. RBD allows the same image to be mapped multiple times on the same node or on different nodes.
addition, I found that if the client is isolated from the network for a long time, the Watcher is released. However, the client still mounts this image. In this situation, if another client can also mount this image and the image is writable from both clients, data corruption occurs. Could you tell me whether this is a realistic scenario?
Yes, this is a realistic scenario which can occur even if the client isn't isolated from the network. If the user does this, it's up to the user to ensure that everything remains consistent. One use case for mapping the same image on multiple nodes is a clustered (also referred to as a shared disk) filesystem, such as OCFS2. Thanks, Ilya
Hi, we run into the same issue and there is actually another use case: live-migration of VMs. This requires an RBD image being mapped to two clients simultaneously, so this is intentional. If multiple clints map an image in RW-mode, the ceph back-end will cycle the write lock between the clients to allow each of them to flush writes, this is intentional. The way to coordinate here is the job of the orchestrator. In this case specifically, its explicitly managing a write lock during live-migration such that writes occur in the correct order. Its not a ceph job, its an orchestration job. The rbd interface just provides the tools to do it, for example, you can attach information that helps you hunting down dead-looking clients and kill them proper before mapping an image somewhere else. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Ilya Dryomov <idryomov@gmail.com> Sent: Thursday, May 23, 2024 2:05 PM To: Yuma Ogami Cc: ceph-users@ceph.io Subject: [ceph-users] Re: does the RBD client block write when the Watcher times out? On Thu, May 23, 2024 at 4:48 AM Yuma Ogami <yuma.ogami.cybozu@gmail.com> wrote:
Hello.
I'm currently verifying the behavior of RBD on failure. I'm wondering about the consistency of RBD images after network failures. As a result of my investigation, I found that RBD sets a Watcher to RBD image if a client mounts this volume to prevent multiple mounts. In
Hi Yuma, The watcher is created to watch for updates (technically, to listen to notifications) on the RBD image, not to prevent multiple mounts. RBD allows the same image to be mapped multiple times on the same node or on different nodes.
addition, I found that if the client is isolated from the network for a long time, the Watcher is released. However, the client still mounts this image. In this situation, if another client can also mount this image and the image is writable from both clients, data corruption occurs. Could you tell me whether this is a realistic scenario?
Yes, this is a realistic scenario which can occur even if the client isn't isolated from the network. If the user does this, it's up to the user to ensure that everything remains consistent. One use case for mapping the same image on multiple nodes is a clustered (also referred to as a shared disk) filesystem, such as OCFS2. Thanks, Ilya _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, I understood that the watcher cannot prevent multiple mounts. Based on the feedback I received, I will consider countermeasures. Thank you for your valuable insights. Yuma. 2024年5月23日(木) 21:15 Frank Schilder <frans@dtu.dk>:
Hi, we run into the same issue and there is actually another use case: live-migration of VMs. This requires an RBD image being mapped to two clients simultaneously, so this is intentional. If multiple clints map an image in RW-mode, the ceph back-end will cycle the write lock between the clients to allow each of them to flush writes, this is intentional. The way to coordinate here is the job of the orchestrator. In this case specifically, its explicitly managing a write lock during live-migration such that writes occur in the correct order.
Its not a ceph job, its an orchestration job. The rbd interface just provides the tools to do it, for example, you can attach information that helps you hunting down dead-looking clients and kill them proper before mapping an image somewhere else.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Ilya Dryomov <idryomov@gmail.com> Sent: Thursday, May 23, 2024 2:05 PM To: Yuma Ogami Cc: ceph-users@ceph.io Subject: [ceph-users] Re: does the RBD client block write when the Watcher times out?
On Thu, May 23, 2024 at 4:48 AM Yuma Ogami <yuma.ogami.cybozu@gmail.com> wrote:
Hello.
I'm currently verifying the behavior of RBD on failure. I'm wondering about the consistency of RBD images after network failures. As a result of my investigation, I found that RBD sets a Watcher to RBD image if a client mounts this volume to prevent multiple mounts. In
Hi Yuma,
The watcher is created to watch for updates (technically, to listen to notifications) on the RBD image, not to prevent multiple mounts. RBD allows the same image to be mapped multiple times on the same node or on different nodes.
addition, I found that if the client is isolated from the network for a long time, the Watcher is released. However, the client still mounts this image. In this situation, if another client can also mount this image and the image is writable from both clients, data corruption occurs. Could you tell me whether this is a realistic scenario?
Yes, this is a realistic scenario which can occur even if the client isn't isolated from the network. If the user does this, it's up to the user to ensure that everything remains consistent. One use case for mapping the same image on multiple nodes is a clustered (also referred to as a shared disk) filesystem, such as OCFS2.
Thanks,
Ilya _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
caskd
-
Frank Schilder
-
Ilya Dryomov
-
Yuma Ogami