15 Jan
2021
15 Jan
'21
9:36 a.m.
Hello cephers,
I run Nautilus (14.2.15)
Here is my context : each night a script take a snapshot from each RBD
volume in a pool (all the disks of the VMs hosted) on my ceph production
cluster. Then each snapshot is exported (rbd export-diff | rbd
import-diff in SSH) towards my ceph bakup cluster.
0. Yesterday, and for the firt time, I had this error when I ran ''rbd
-p MY_POOL du'' on the ceph backup cluster :
021-01-14 11:41:29.294 7f1f2fbcf0c0 -1 librbd::DiffIterate:
diff_object_map: failed to load object map
*rbd_object_map.781776950f8dd5.000000000000b740*
1. I tried to find the image concerned by the error, and I find this
one, with a slight diffence with the number of the object-map :
rbd image 'MY_IMAGE':
size 30 GiB in 7680 objects
order 22 (4 MiB objects)
snapshot_count: 32
id: 781776950f8dd5
block_name_prefix: *rbd_data.781776950f8dd5*
format: 2
features: layering, exclusive-lock, object-map, fast-diff,
deep-flatten, operations
op_features: snap-trash
flags:
create_timestamp: Thu Mar 19 01:00:27 2020
access_timestamp: Thu Mar 19 01:00:27 2020
modify_timestamp: Thu Jan 14 01:42:23 2021
2. Then I ran a check wich not show any error
rbd -p MY_POOL object-map check "MY_IMAGE"
Object Map Check: 100% complete...done.
3. Then, I still ran a rebuild
rbd -p MY_POOL object-map rebuild "MY_IMAGE"
Object Map Rebuild: 100% complete...done.
4. But the error is still here :
2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate:
diff_object_map: failed to load object map
rbd_object_map.781776950f8dd5.000000000000b740
Do someone know how to find the the image concerned by the object-map
error and fix this error.
Thank you for your help,
Rafael
15 Jan
15 Jan
9:44 a.m.
Hi,
can you stat the object? If yes, does it have a newer timestamp from
your rebuild?
rados -p <pool> stat rbd_object_map.781776950f8dd5.000000000000b740
Zitat von Rafael Diaz Maurin <Rafael.DiazMaurin(a)univ-rennes1.fr>fr>:
Hello cephers,
I run Nautilus (14.2.15)
Here is my context : each night a script take a snapshot from each
RBD volume in a pool (all the disks of the VMs hosted) on my ceph
production cluster. Then each snapshot is exported (rbd export-diff
| rbd import-diff in SSH) towards my ceph bakup cluster.
0. Yesterday, and for the firt time, I had this error when I ran
''rbd -p MY_POOL du'' on the ceph backup cluster :
021-01-14 11:41:29.294 7f1f2fbcf0c0 -1 librbd::DiffIterate:
diff_object_map: failed to load object map
*rbd_object_map.781776950f8dd5.000000000000b740*
1. I tried to find the image concerned by the error, and I find this
one, with a slight diffence with the number of the object-map :
rbd image 'MY_IMAGE':
size 30 GiB in 7680 objects
order 22 (4 MiB objects)
snapshot_count: 32
id: 781776950f8dd5
block_name_prefix: *rbd_data.781776950f8dd5*
format: 2
features: layering, exclusive-lock, object-map, fast-diff,
deep-flatten, operations
op_features: snap-trash
flags:
create_timestamp: Thu Mar 19 01:00:27 2020
access_timestamp: Thu Mar 19 01:00:27 2020
modify_timestamp: Thu Jan 14 01:42:23 2021
2. Then I ran a check wich not show any error
rbd -p MY_POOL object-map check "MY_IMAGE"
Object Map Check: 100% complete...done.
3. Then, I still ran a rebuild
rbd -p MY_POOL object-map rebuild "MY_IMAGE"
Object Map Rebuild: 100% complete...done.
4. But the error is still here :
2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate:
diff_object_map: failed to load object map
rbd_object_map.781776950f8dd5.000000000000b740
Do someone know how to find the the image concerned by the
object-map error and fix this error.
Thank you for your help,
Rafael
2:39 p.m.
On Fri, Jan 15, 2021 at 4:36 AM Rafael Diaz Maurin
<Rafael.DiazMaurin(a)univ-rennes1.fr> wrote:
Hello cephers,
I run Nautilus (14.2.15)
Here is my context : each night a script take a snapshot from each RBD volume in a pool
(all the disks of the VMs hosted) on my ceph production cluster. Then each snapshot is
exported (rbd export-diff | rbd import-diff in SSH) towards my ceph bakup cluster.
0. Yesterday, and for the firt time, I had this error when I ran ''rbd -p MY_POOL
du'' on the ceph backup cluster :
021-01-14 11:41:29.294 7f1f2fbcf0c0 -1 librbd::DiffIterate: diff_object_map: failed to
load object map rbd_object_map.781776950f8dd5.000000000000b740
1. I tried to find the image concerned by the error, and I find this one, with a slight
diffence with the number of the object-map :
rbd image 'MY_IMAGE':
size 30 GiB in 7680 objects
order 22 (4 MiB objects)
snapshot_count: 32
id: 781776950f8dd5
block_name_prefix: rbd_data.781776950f8dd5
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, operations
op_features: snap-trash
flags:
create_timestamp: Thu Mar 19 01:00:27 2020
access_timestamp: Thu Mar 19 01:00:27 2020
modify_timestamp: Thu Jan 14 01:42:23 2021
2. Then I ran a check wich not show any error
rbd -p MY_POOL object-map check "MY_IMAGE"
Object Map Check: 100% complete...done.
3. Then, I still ran a rebuild
rbd -p MY_POOL object-map rebuild "MY_IMAGE"
Object Map Rebuild: 100% complete...done.
4. But the error is still here :
2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to
load object map rbd_object_map.781776950f8dd5.000000000000b740
You would need to rebuild the snapshot object map. First find the name
of the snapshot associated with snapshot id b740 (46912) via "rbd snap
ls <image-spec>", then run rebuild on that particule snapshot via "rbd
object-map rebuild <image-spec>@<snap name>".
Do someone know how to find the the image concerned by the object-map error and fix this
error.
Thank you for your help,
Rafael
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
3:11 p.m.
Le 15/01/2021 à 15:39, Jason Dillaman a écrit :
Unfortunately it did not work : I can't succeed in finding the broken
snaphot.
I have parsed all the snaphots of all the images in a loop, where I get
all the SNAPIDs.
But none of these IDs is *46912* !
Please, can you tell me how did you get this number *46912* or a link
which explain that ?
Many thanks,
Rafael
--
Rafael Diaz Maurin
DSI de l'Université de Rennes 1
Pôle Infrastructures, équipe Systèmes
02 23 23 71 57
4. But the
error is still here :
2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to
load object map rbd_object_map.781776950f8dd5.000000000000b740
You would need to
rebuild the snapshot object map. First find the name
of the snapshot associated with snapshot id b740 (46912) via "rbd snap
ls <image-spec>", then run rebuild on that particule snapshot via "rbd
object-map rebuild <image-spec>@<snap name>".
3:29 p.m.
On Fri, Jan 15, 2021 at 10:12 AM Rafael Diaz Maurin
<Rafael.DiazMaurin(a)univ-rennes1.fr> wrote:
Le 15/01/2021 à 15:39, Jason Dillaman a écrit :
4. But the error is still here :
2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to
load object map rbd_object_map.781776950f8dd5.000000000000b740
You would need to rebuild the snapshot object map. First find the name
of the snapshot associated with snapshot id b740 (46912) via "rbd snap
ls <image-spec>", then run rebuild on that particule snapshot via "rbd
object-map rebuild <image-spec>@<snap name>".
Unfortunately it did not work : I can't succeed in finding the broken snaphot.
I have parsed all the snaphots of all the images in a loop, where I get all the SNAPIDs.
But none of these IDs is 46912 !
"
Please, can you tell me how did you get this number 46912 or a link which explain that ?
You wouldn't need to search all images, it's tied to image "MY_IMAGE".
Have you tried "rbd snap ls --all MY_IMAGE"?
Many thanks,
Rafael
--
Rafael Diaz Maurin
DSI de l'Université de Rennes 1
Pôle Infrastructures, équipe Systèmes
02 23 23 71 57
--
Jason
3:49 p.m.
Le 15/01/2021 à 16:29, Jason Dillaman a écrit :
On Fri, Jan 15, 2021 at 10:12 AM Rafael Diaz Maurin
<Rafael.DiazMaurin(a)univ-rennes1.fr> wrote:
Yes : now I got the broken snapshot with the --all flag.
In fact this snapshot was in the namespace "trash" that's why I couldn't
find it with "rbd snap ls"
@list : the id b740 is hexadecimal, we need to convert it in decimal in
order to get the id 46912
Thanks a lot for you answers and for the real good job done on Ceph !
Rafael
Le 15/01/2021 à 15:39, Jason Dillaman a écrit :
4. But the error is still here :
2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to
load object map rbd_object_map.781776950f8dd5.000000000000b740
You would need to rebuild the snapshot object map. First find the name
of the snapshot associated with snapshot id b740 (46912) via "rbd snap
ls <image-spec>", then run rebuild on that particule snapshot via "rbd
object-map rebuild <image-spec>@<snap name>".
Unfortunately it did not work : I can't succeed in finding the broken snaphot.
I have parsed all the snaphots of all the images in a loop, where I get all the SNAPIDs.
But none of these IDs is 46912 !
"
Please, can you tell me how did you get this number 46912 or a link which explain that ?
You wouldn't need to search all images, it's tied to image
"MY_IMAGE".
Have you tried "rbd snap ls --all MY_IMAGE"?
1220
days inactive
1220
days old
5 comments
3 participants
participants (3)
-
Eugen Block -
Jason Dillaman -
Rafael Diaz Maurin