RBD Mirror - Failed to unlink peer
Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.
Hi, I don't have much to contribute, but according to the source code [1] this seems to be a non-fatal message: void CreatePrimaryRequest<I>::handle_unlink_peer(int r) { CephContext *cct = m_image_ctx->cct; ldout(cct, 15) << "r=" << r << dendl; if (r < 0) { lderr(cct) << "failed to unlink peer: " << cpp_strerror(r) << dendl; finish(0); // not fatal return; } I guess if you increased debug level to 15, you might see where exactly that message comes from. But I don't know how to get rid of them, so maybe one of the devs can comment on that. Regards, Eugen [1] https://github.com/ceph/ceph/blob/v17.2.7/src/librbd/mirror/snapshot/CreateP... Zitat von Scott Cairns <Scott.Cairns@tecnica-ltd.co.uk>:
Hi,
Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot.
We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error:
librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory
This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot.
What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does.
I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot.
I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again.
Any ideas?
Thanks, Scott
________________________________
The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited.
Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42.
This footnote also confirms that this email message has been swept for the presence of computer viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks - hopefully I'll hear back from devs then as I can't seem to find anything online about others encountering the same warning, but I surely can't be the only one! Would it be the rbd subsystem I'm looking to increase to debug level 15 or is there another subsystem for rbd mirroring? What would be the best way to enable it (ceph config set client debug_rbd 20 then change back to 0/5 once done)?
Hi, sorry for the delayed response, I was on vacation. I would set the "debug_rbd_mirror" config to 15 (or higher) and then watch the logs: # ceph config set client.rbd-mirror.<MY_MIRROR> debug_rbd_mirror 15 Maybe that reveals anything. Regards, Eugen Zitat von scott.cairns@tecnica-ltd.co.uk:
Thanks - hopefully I'll hear back from devs then as I can't seem to find anything online about others encountering the same warning, but I surely can't be the only one!
Would it be the rbd subsystem I'm looking to increase to debug level 15 or is there another subsystem for rbd mirroring? What would be the best way to enable it (ceph config set client debug_rbd 20 then change back to 0/5 once done)? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks - side tracked with other work so only just got around to testing this. Unfortunately when enabling rbd-mirror logs on the source cluster I'm not seeing any events logged at all, however on the remote cluster I can see constant activity (mostly imageReplayer, mirrorStatusUpdater, etc. logs). Currently our sync is only one way (from source to remote), and the error appears to be on the source (i.e. as soon as the snapshot is taken). There's no error on the remote cluster in the rbd mirror logs, and nothing logged at all on the source cluster in the rbd mirror logs.
Hi, I think I need some clarification. You have a rbd-mirror daemon running on the primary site although you have configured rbd-mirroring one-way only? And those errors you see in the rbd-mirror daemon log on the primary site? Maybe the daemon got started/activated by accident (or it was not disabled from some two-way mirror tests)? You don't need a rbd-mirror daemon on the primary site if you mirror only one-way. Zitat von scott.cairns@tecnica-ltd.co.uk:
Thanks - side tracked with other work so only just got around to testing this.
Unfortunately when enabling rbd-mirror logs on the source cluster I'm not seeing any events logged at all, however on the remote cluster I can see constant activity (mostly imageReplayer, mirrorStatusUpdater, etc. logs).
Currently our sync is only one way (from source to remote), and the error appears to be on the source (i.e. as soon as the snapshot is taken).
There's no error on the remote cluster in the rbd mirror logs, and nothing logged at all on the source cluster in the rbd mirror logs. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We have rbd-mirror daemon running on both sites, however replication is only one way (i.e. the one on the remote site is the only live one, the one on the primary site is just there for if we ever need to set up two-way, but this is not currently set up for any replication - so it makes sense there's nothing in the log files on the primary site, as it's doing nothing). I'm not seeing any errors in rbd-mirror daemon log at either end - primary is blank as expected, and the error appears to be on the primary when the snapshot is taken, so the remote cluster never see's any errors. When we either manually run the command to take a snapshot, or have this run through cron we receive the error, e.g. running the following on the primary site: # rbd mirror image snapshot ceph-ssd/vm-101-disk-1 Snapshot ID: 58393 2024-08-26T12:39:54.958+0100 7b5ad6a006c0 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7b5ac0019e60 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This appears in the console as the output for this (we used to only get the Snapshot ID: xxxxx), not in any rbd log files. Hope that clarifies it? Thanks.
Can you share 'ceph versions' output? Do you see the same behaviour when adding a snapshot schedule, e.g. rbd -p <pool> mirror snapshot schedule add 30m I can't reproduce it, unfortunately, creating those mirror snapshots manually still works for me. Zitat von scott.cairns@tecnica-ltd.co.uk:
We have rbd-mirror daemon running on both sites, however replication is only one way (i.e. the one on the remote site is the only live one, the one on the primary site is just there for if we ever need to set up two-way, but this is not currently set up for any replication - so it makes sense there's nothing in the log files on the primary site, as it's doing nothing).
I'm not seeing any errors in rbd-mirror daemon log at either end - primary is blank as expected, and the error appears to be on the primary when the snapshot is taken, so the remote cluster never see's any errors.
When we either manually run the command to take a snapshot, or have this run through cron we receive the error, e.g. running the following on the primary site:
# rbd mirror image snapshot ceph-ssd/vm-101-disk-1 Snapshot ID: 58393 2024-08-26T12:39:54.958+0100 7b5ad6a006c0 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7b5ac0019e60 handle_unlink_peer: failed to unlink peer: (2) No such file or directory
This appears in the console as the output for this (we used to only get the Snapshot ID: xxxxx), not in any rbd log files.
Hope that clarifies it? Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey! Were you ever able to resolve this? We came across the same issue in Ceph 18.2.4 - tried several options (recreating the mirror from scratch, disable and enable, change the scheduling) but the error messages still appear - despite everything looking ok. We are lso not able to reproduce the issue in a test enviroment. Regards Kevin
participants (4)
-
Eugen Block
-
Kevin Schneider
-
Scott Cairns
-
scott.cairns＠tecnica-ltd.co.uk