librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.
Hello cephers, I run Nautilus (14.2.15) Here is my context : each night a script take a snapshot from each RBD volume in a pool (all the disks of the VMs hosted) on my ceph production cluster. Then each snapshot is exported (rbd export-diff | rbd import-diff in SSH) towards my ceph bakup cluster. 0. Yesterday, and for the firt time, I had this error when I ran ''rbd -p MY_POOL du'' on the ceph backup cluster : 021-01-14 11:41:29.294 7f1f2fbcf0c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map *rbd_object_map.781776950f8dd5.000000000000b740* 1. I tried to find the image concerned by the error, and I find this one, with a slight diffence with the number of the object-map : rbd image 'MY_IMAGE': size 30 GiB in 7680 objects order 22 (4 MiB objects) snapshot_count: 32 id: 781776950f8dd5 block_name_prefix: *rbd_data.781776950f8dd5* format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, operations op_features: snap-trash flags: create_timestamp: Thu Mar 19 01:00:27 2020 access_timestamp: Thu Mar 19 01:00:27 2020 modify_timestamp: Thu Jan 14 01:42:23 2021 2. Then I ran a check wich not show any error rbd -p MY_POOL object-map check "MY_IMAGE" Object Map Check: 100% complete...done. 3. Then, I still ran a rebuild rbd -p MY_POOL object-map rebuild "MY_IMAGE" Object Map Rebuild: 100% complete...done. 4. But the error is still here : 2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740 Do someone know how to find the the image concerned by the object-map error and fix this error. Thank you for your help, Rafael
Hi, can you stat the object? If yes, does it have a newer timestamp from your rebuild? rados -p <pool> stat rbd_object_map.781776950f8dd5.000000000000b740 Zitat von Rafael Diaz Maurin <Rafael.DiazMaurin@univ-rennes1.fr>:
Hello cephers,
I run Nautilus (14.2.15)
Here is my context : each night a script take a snapshot from each RBD volume in a pool (all the disks of the VMs hosted) on my ceph production cluster. Then each snapshot is exported (rbd export-diff | rbd import-diff in SSH) towards my ceph bakup cluster.
0. Yesterday, and for the firt time, I had this error when I ran ''rbd -p MY_POOL du'' on the ceph backup cluster : 021-01-14 11:41:29.294 7f1f2fbcf0c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map *rbd_object_map.781776950f8dd5.000000000000b740*
1. I tried to find the image concerned by the error, and I find this one, with a slight diffence with the number of the object-map :
rbd image 'MY_IMAGE': size 30 GiB in 7680 objects order 22 (4 MiB objects) snapshot_count: 32 id: 781776950f8dd5 block_name_prefix: *rbd_data.781776950f8dd5* format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, operations op_features: snap-trash flags: create_timestamp: Thu Mar 19 01:00:27 2020 access_timestamp: Thu Mar 19 01:00:27 2020 modify_timestamp: Thu Jan 14 01:42:23 2021
2. Then I ran a check wich not show any error rbd -p MY_POOL object-map check "MY_IMAGE" Object Map Check: 100% complete...done.
3. Then, I still ran a rebuild rbd -p MY_POOL object-map rebuild "MY_IMAGE" Object Map Rebuild: 100% complete...done.
4. But the error is still here : 2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740
Do someone know how to find the the image concerned by the object-map error and fix this error.
Thank you for your help,
Rafael
On Fri, Jan 15, 2021 at 4:36 AM Rafael Diaz Maurin <Rafael.DiazMaurin@univ-rennes1.fr> wrote:
Hello cephers,
I run Nautilus (14.2.15)
Here is my context : each night a script take a snapshot from each RBD volume in a pool (all the disks of the VMs hosted) on my ceph production cluster. Then each snapshot is exported (rbd export-diff | rbd import-diff in SSH) towards my ceph bakup cluster.
0. Yesterday, and for the firt time, I had this error when I ran ''rbd -p MY_POOL du'' on the ceph backup cluster : 021-01-14 11:41:29.294 7f1f2fbcf0c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740
1. I tried to find the image concerned by the error, and I find this one, with a slight diffence with the number of the object-map :
rbd image 'MY_IMAGE': size 30 GiB in 7680 objects order 22 (4 MiB objects) snapshot_count: 32 id: 781776950f8dd5 block_name_prefix: rbd_data.781776950f8dd5 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, operations op_features: snap-trash flags: create_timestamp: Thu Mar 19 01:00:27 2020 access_timestamp: Thu Mar 19 01:00:27 2020 modify_timestamp: Thu Jan 14 01:42:23 2021
2. Then I ran a check wich not show any error rbd -p MY_POOL object-map check "MY_IMAGE" Object Map Check: 100% complete...done.
3. Then, I still ran a rebuild rbd -p MY_POOL object-map rebuild "MY_IMAGE" Object Map Rebuild: 100% complete...done.
4. But the error is still here : 2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740
You would need to rebuild the snapshot object map. First find the name of the snapshot associated with snapshot id b740 (46912) via "rbd snap ls <image-spec>", then run rebuild on that particule snapshot via "rbd object-map rebuild <image-spec>@<snap name>".
Do someone know how to find the the image concerned by the object-map error and fix this error.
Thank you for your help,
Rafael
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jason
Le 15/01/2021 à 15:39, Jason Dillaman a écrit :
4. But the error is still here : 2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740 You would need to rebuild the snapshot object map. First find the name of the snapshot associated with snapshot id b740 (46912) via "rbd snap ls <image-spec>", then run rebuild on that particule snapshot via "rbd object-map rebuild <image-spec>@<snap name>".
Unfortunately it did not work : I can't succeed in finding the broken snaphot. I have parsed all the snaphots of all the images in a loop, where I get all the SNAPIDs. But none of these IDs is *46912* ! Please, can you tell me how did you get this number *46912* or a link which explain that ? Many thanks, Rafael -- Rafael Diaz Maurin DSI de l'Université de Rennes 1 Pôle Infrastructures, équipe Systèmes 02 23 23 71 57
On Fri, Jan 15, 2021 at 10:12 AM Rafael Diaz Maurin <Rafael.DiazMaurin@univ-rennes1.fr> wrote:
Le 15/01/2021 à 15:39, Jason Dillaman a écrit :
4. But the error is still here : 2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740
You would need to rebuild the snapshot object map. First find the name of the snapshot associated with snapshot id b740 (46912) via "rbd snap ls <image-spec>", then run rebuild on that particule snapshot via "rbd object-map rebuild <image-spec>@<snap name>".
Unfortunately it did not work : I can't succeed in finding the broken snaphot.
I have parsed all the snaphots of all the images in a loop, where I get all the SNAPIDs.
But none of these IDs is 46912 ! " Please, can you tell me how did you get this number 46912 or a link which explain that ?
You wouldn't need to search all images, it's tied to image "MY_IMAGE". Have you tried "rbd snap ls --all MY_IMAGE"?
Many thanks,
Rafael
-- Rafael Diaz Maurin DSI de l'Université de Rennes 1 Pôle Infrastructures, équipe Systèmes 02 23 23 71 57
-- Jason
Le 15/01/2021 à 16:29, Jason Dillaman a écrit :
On Fri, Jan 15, 2021 at 10:12 AM Rafael Diaz Maurin <Rafael.DiazMaurin@univ-rennes1.fr> wrote:
Le 15/01/2021 à 15:39, Jason Dillaman a écrit :
4. But the error is still here : 2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740
You would need to rebuild the snapshot object map. First find the name of the snapshot associated with snapshot id b740 (46912) via "rbd snap ls <image-spec>", then run rebuild on that particule snapshot via "rbd object-map rebuild <image-spec>@<snap name>".
Unfortunately it did not work : I can't succeed in finding the broken snaphot.
I have parsed all the snaphots of all the images in a loop, where I get all the SNAPIDs.
But none of these IDs is 46912 ! " Please, can you tell me how did you get this number 46912 or a link which explain that ? You wouldn't need to search all images, it's tied to image "MY_IMAGE". Have you tried "rbd snap ls --all MY_IMAGE"?
Yes : now I got the broken snapshot with the --all flag. In fact this snapshot was in the namespace "trash" that's why I couldn't find it with "rbd snap ls" @list : the id b740 is hexadecimal, we need to convert it in decimal in order to get the id 46912 Thanks a lot for you answers and for the real good job done on Ceph ! Rafael
participants (3)
-
Eugen Block
-
Jason Dillaman
-
Rafael Diaz Maurin