rbd export-diff/import-diff hangs
Hi, I'm using rbd import and export to copy image from one cluster to another. Also using import-diff and export-diff to update image in remote cluster. For example, "rbd --cluster local export-diff ... | rbd --cluster remote import-diff ...". Sometimes, the whole command is stuck. I can't tell it's stuck on which end of the pipe. I did some search, [1] seems the same issue and [2] is also related. Wonder if there is any way to identify where it's stuck and get more debugging info. Given [2], I'd suspect the import-diff is stuck, cause rbd client is importing to the remote cluster. Networking latency could be involved here? Ping latency is 7~8 ms. Any comments is appreciated! [1] https://bugs.launchpad.net/cinder/+bug/2031897 [2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs Thanks! Tony
To update, hanging happens when updating local image, not remote, networking is not a concern here. Any advices how to look into it? Thanks! Tony ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: August 26, 2023 10:43 PM To: dev@ceph.io; ceph-users@ceph.io Subject: [ceph-users] rbd export-diff/import-diff hangs Hi, I'm using rbd import and export to copy image from one cluster to another. Also using import-diff and export-diff to update image in remote cluster. For example, "rbd --cluster local export-diff ... | rbd --cluster remote import-diff ...". Sometimes, the whole command is stuck. I can't tell it's stuck on which end of the pipe. I did some search, [1] seems the same issue and [2] is also related. Wonder if there is any way to identify where it's stuck and get more debugging info. Given [2], I'd suspect the import-diff is stuck, cause rbd client is importing to the remote cluster. Networking latency could be involved here? Ping latency is 7~8 ms. Any comments is appreciated! [1] https://bugs.launchpad.net/cinder/+bug/2031897 [2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You mean the image is in use while you’re exporting? Have thought about creating snapshots and exporting those? Or set up rbd mirroring? Zitat von Tony Liu <tonyliu0592@hotmail.com>:
To update, hanging happens when updating local image, not remote, networking is not a concern here. Any advices how to look into it?
Thanks! Tony ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: August 26, 2023 10:43 PM To: dev@ceph.io; ceph-users@ceph.io Subject: [ceph-users] rbd export-diff/import-diff hangs
Hi,
I'm using rbd import and export to copy image from one cluster to another. Also using import-diff and export-diff to update image in remote cluster. For example, "rbd --cluster local export-diff ... | rbd --cluster remote import-diff ...". Sometimes, the whole command is stuck. I can't tell it's stuck on which end of the pipe. I did some search, [1] seems the same issue and [2] is also related.
Wonder if there is any way to identify where it's stuck and get more debugging info. Given [2], I'd suspect the import-diff is stuck, cause rbd client is importing to the remote cluster. Networking latency could be involved here? Ping latency is 7~8 ms.
Any comments is appreciated!
[1] https://bugs.launchpad.net/cinder/+bug/2031897 [2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs
Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It's export-diff from an in-use image, both from-snapshot and to-snapshot exist. The same from-snapshot exists in import image, which is the to-snapshot from last diff. export/import is used for local backup, rbd-mirroring is used for remote backup. Looking for options to get more info to troubleshoot. Thanks! Tony ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: August 27, 2023 11:53 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: rbd export-diff/import-diff hangs You mean the image is in use while you’re exporting? Have thought about creating snapshots and exporting those? Or set up rbd mirroring? Zitat von Tony Liu <tonyliu0592@hotmail.com>:
To update, hanging happens when updating local image, not remote, networking is not a concern here. Any advices how to look into it?
Thanks! Tony ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: August 26, 2023 10:43 PM To: dev@ceph.io; ceph-users@ceph.io Subject: [ceph-users] rbd export-diff/import-diff hangs
Hi,
I'm using rbd import and export to copy image from one cluster to another. Also using import-diff and export-diff to update image in remote cluster. For example, "rbd --cluster local export-diff ... | rbd --cluster remote import-diff ...". Sometimes, the whole command is stuck. I can't tell it's stuck on which end of the pipe. I did some search, [1] seems the same issue and [2] is also related.
Wonder if there is any way to identify where it's stuck and get more debugging info. Given [2], I'd suspect the import-diff is stuck, cause rbd client is importing to the remote cluster. Networking latency could be involved here? Ping latency is 7~8 ms.
Any comments is appreciated!
[1] https://bugs.launchpad.net/cinder/+bug/2031897 [2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs
Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Figured it out. It's not rbd issue. Sorry for this false alarm. Thanks! Tony ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: August 27, 2023 08:19 PM To: Eugen Block; ceph-users@ceph.io Subject: [ceph-users] Re: rbd export-diff/import-diff hangs It's export-diff from an in-use image, both from-snapshot and to-snapshot exist. The same from-snapshot exists in import image, which is the to-snapshot from last diff. export/import is used for local backup, rbd-mirroring is used for remote backup. Looking for options to get more info to troubleshoot. Thanks! Tony ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: August 27, 2023 11:53 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: rbd export-diff/import-diff hangs You mean the image is in use while you’re exporting? Have thought about creating snapshots and exporting those? Or set up rbd mirroring? Zitat von Tony Liu <tonyliu0592@hotmail.com>:
To update, hanging happens when updating local image, not remote, networking is not a concern here. Any advices how to look into it?
Thanks! Tony ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: August 26, 2023 10:43 PM To: dev@ceph.io; ceph-users@ceph.io Subject: [ceph-users] rbd export-diff/import-diff hangs
Hi,
I'm using rbd import and export to copy image from one cluster to another. Also using import-diff and export-diff to update image in remote cluster. For example, "rbd --cluster local export-diff ... | rbd --cluster remote import-diff ...". Sometimes, the whole command is stuck. I can't tell it's stuck on which end of the pipe. I did some search, [1] seems the same issue and [2] is also related.
Wonder if there is any way to identify where it's stuck and get more debugging info. Given [2], I'd suspect the import-diff is stuck, cause rbd client is importing to the remote cluster. Networking latency could be involved here? Ping latency is 7~8 ms.
Any comments is appreciated!
[1] https://bugs.launchpad.net/cinder/+bug/2031897 [2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs
Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, Aug 28, 2023 at 6:21 AM Tony Liu <tonyliu0592@hotmail.com> wrote:
It's export-diff from an in-use image, both from-snapshot and to-snapshot exist. The same from-snapshot exists in import image, which is the to-snapshot from last diff. export/import is used for local backup, rbd-mirroring is used for remote backup.
Just to make it clear, do you mean you are running export-diff for an image that is being mirrored (snapshot based)?
Looking for options to get more info to troubleshoot.
I would split "rbd export-diff | rbd import-diff" into two commands: rbd export-diff > image.diff rbd import-diff < image.diff and see if it gets stuck for the first one, so we are sure the export-diff is the issue here. The next step would be enabling rbd debug, something like this: rbd export-diff .. --debug-rbd=20 --log-file=/tmp/{image}_{from_snap}_{to_snap}.log --log-to-stderr=false Hope it will not use too much space and you will be able to get a log for a getting stuck case. Then please provide the log for review somehow. Also, notice the time when you interrupt the hanging export-diff. -- Mykola Golub
participants (3)
-
Eugen Block
-
Mykola Golub
-
Tony Liu