Hi, I appear to have found the patch where the behaviour of the 'whole-object' switch changed when listing the differences between two snapshots. Various backups tools (eg backy2 and benji) rely on JSON diff output to perform incremental Ceph image backups by listing the differences between two RBD snapshots. The 'whole-object' parameter is supposed to limit these differences to the 4 MiB objects that make up images. The following issue's resulting patch alters the behaviour to list all objects that the snapshot refers to, but only when adding the 'whole-object' parameter: https://tracker.ceph.com/issues/42248 In my earlier mail I demonstrate that I write 64 MiB to a new image, create the first snapshot and then overwrite the first 4 MiB before creating a second snapshot. The differences should always only be the first 4 MiB object, irrespective whether or not 'whole-object' is passed, but the result is that everything is perfect without the parameter whilst it lists all 64 MiB allocated space when the switch is added. PS: I corrected the line breaks which would have made my previous email very difficult to read... Regards David Herselman -----Original Message----- From: David Herselman &lt;dhe(a)syrex.co&gt; Sent: Wednesday, 12 May 2021 11:55 PM To: ceph-users(a)ceph.io Subject: [ceph-users] Ceph Octopus 15.2.11 - rbd diff --from-snap lists all objects Hi, Has something change with 'rbd diff' in Octopus or have I hit a bug? I am no longer able to obtain the list of objects that have changed between two snapshots of an image, it always lists all allocated regions of the RBD image. This behaviour however only occurs when I add the '--whole-object' switch. Using KRBD client with kernel 5.11.7 and Ceph Octopus 15.2.11 as part of Proxmox PVE 6.4 which is based on Debian 10. Images have the following features and I've performed offline object map checks and rebuilds (no errors reported). To reproduce my issue I first create a new RBD image (default features are 63), map it using KRBD, write some data, create first snapshot, write a single object (4 MiB), create a second snapshot and then list the differences: [admin@kvm1a ~]# rbd create rbd_hdd/test --size 40G [admin@kvm1a ~]# rbd info rbd_hdd/test rbd image 'test': size 40 GiB in 10240 objects order 22 (4 MiB objects) snapshot_count: 0 id: 73363f8443987b block_name_prefix: rbd_data.73363f8443987b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Wed May 12 23:01:11 2021 access_timestamp: Wed May 12 23:01:11 2021 modify_timestamp: Wed May 12 23:01:11 2021 [admin@kvm1a ~]# rbd map rbd_hdd/test /dev/rbd18 [admin@kvm1a ~]# dd if=/dev/zero of=/dev/rbd18 bs=64M count=1 1+0 records in 1+0 records out 67108864 bytes (67 MB, 64 MiB) copied, 0.668701 s, 100 MB/s [admin@kvm1a ~]# sync [admin@kvm1a ~]# rbd snap create rbd_hdd/test@snap1 [admin@kvm1a ~]# dd if=/dev/zero of=/dev/rbd18 bs=4M count=1 1+0 records in 1+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.265691 s, 15.8 MB/s [admin@kvm1a ~]# sync [admin@kvm1a ~]# rbd snap create rbd_hdd/test@snap2 [admin@kvm1a ~]# rbd diff --from-snap snap1 rbd_hdd/test@snap2 --format=json [{"offset":0,"length":4194304,"exists":"true"}] [admin@kvm1b ~]# rbd diff --from-snap snap1 rbd_hdd/test@snap2 --format=json --whole-object [{"offset":0,"length":4194304,"exists":"true"}, {"offset":4194304,"length":4194304,"exists":"true"}, {"offset":8388608,"length":4194304,"exists":"true"}, {"offset":12582912,"length":4194304,"exists":"true"}, {"offset":16777216,"length":4194304,"exists":"true"}, {"offset":20971520,"length":4194304,"exists":"true"}, {"offset":25165824,"length":4194304,"exists":"true"}, {"offset":29360128,"length":4194304,"exists":"true"}, {"offset":33554432,"length":4194304,"exists":"true"}, {"offset":37748736,"length":4194304,"exists":"true"}, {"offset":41943040,"length":4194304,"exists":"true"}, {"offset":46137344,"length":4194304,"exists":"true"}, {"offset":50331648,"length":4194304,"exists":"true"}, {"offset":54525952,"length":4194304,"exists":"true"}, {"offset":58720256,"length":4194304,"exists":"true"}, {"offset":62914560,"length":4194304,"exists":"true"}] [admin@kvm1a ~]# rbd du rbd_hdd/test NAME PROVISIONED USED test@snap1 40 GiB 64 MiB test@snap2 40 GiB 64 MiB test 40 GiB 4 MiB <TOTAL> 40 GiB 132 MiB My tests appear to confirm that adding the 'whole-object' option to rbd diff results in it listing every allocated extend instead of only the changes... Regards David Herselman _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io