Running fstrim (discard) inside KVM machine with RBD as disk device corrupts ext4 filesystem
3 Aug
2020
3 Aug
'20
5:10 a.m.
Hey Ceph users,
we are currently facing some serious problems on our Ceph Cluster with
libvirt (KVM), RBD devices and FSTRIM running inside VMs.
The problem is right after running the fstrim command inside the VM the
ext4 filesystem is corrupted and read-only with the following error message:
EXT4-fs error (device sda1): ext4_mb_generate_buddy:756: group 136,
block bitmap and bg descriptor inconsistent: 32200 vs 32768 free clusters
Aborting journal on device sda1-8
EXT4-fs (sda1): Remounting filesystem read-only
EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected
aborted journal
EXT4-fs (sda1): Remounting filesystem read-only
This behavior is reproducible across several VMs with different OS
(Ubuntu 14.04, 16.04 and 18.04) so we guess it is a bug or a
configuration problem regarding RBD devices.
Our setup on the hosts running the VMs looks like:
# lsb_release -d
Description: Ubuntu 20.04 LTS
# uname -a
Linux XXX 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
# ceph --version
ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus
(stable)
-> I know there's the update to Ceph 15.2.4 but I haven't seen any
fstrim/discard related changes in the changelog. If we could fix the
problem with 15.2.4 I would be happy...
The libvirt config for the RBD device with supporting fstrim (discard)
is the following:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='directsync'
io='native'
discard='unmap'/>
<auth username='libvirt'>
<secret type='ceph' usage='client.libvirt'/>
</auth>
<source protocol='rbd' name='cephstorage/testtrim_system'>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
</source>
<target dev='sda' bus='scsi'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
The ceph docs (https://docs.ceph.com/docs/octopus/rbd/qemu-rbd/) gave me
some hints about enabling trim/discard and I tested using 4M as discard
granularity, but I got the same error resulting in a corrupted ext4 file
system.
Changes made to the libvirt config:
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.scsi0-0-0-0.discard_granularity=4194304'/>
</qemu:commandline>
As the RBD devices are thin-provisioned we really need calling fstrim
inside the VM regularly to free up unused blocks, otherwise our Ceph
pool will run out of space.
Any ideas what could be wrong with our RBD setup or can somebody else
reproduce the problem?
Any hints on how to debug this problem?
Any related/open Ceph issues? (I could not fined one)
Thanks a lot for your help, Georg
3 Aug
3 Aug
9:56 a.m.
On Mon, Aug 3, 2020 at 4:11 AM Georg Schönberger
<g.schoenberger(a)xortex.com> wrote:
Hey Ceph users,
we are currently facing some serious problems on our Ceph Cluster with
libvirt (KVM), RBD devices and FSTRIM running inside VMs.
The problem is right after running the fstrim command inside the VM the
ext4 filesystem is corrupted and read-only with the following error message:
EXT4-fs error (device sda1): ext4_mb_generate_buddy:756: group 136,
block bitmap and bg descriptor inconsistent: 32200 vs 32768 free clusters
Aborting journal on device sda1-8
EXT4-fs (sda1): Remounting filesystem read-only
EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected
aborted journal
EXT4-fs (sda1): Remounting filesystem read-only
This behavior is reproducible across several VMs with different OS
(Ubuntu 14.04, 16.04 and 18.04) so we guess it is a bug or a
configuration problem regarding RBD devices.
Our setup on the hosts running the VMs looks like:
# lsb_release -d
Description: Ubuntu 20.04 LTS
# uname -a
Linux XXX 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
# ceph --version
ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus
(stable)
-> I know there's the update to Ceph 15.2.4 but I haven't seen any
fstrim/discard related changes in the changelog. If we could fix the
problem with 15.2.4 I would be happy...
The libvirt config for the RBD device with supporting fstrim (discard)
is the following:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='directsync'
io='native'
discard='unmap'/>
<auth username='libvirt'>
<secret type='ceph' usage='client.libvirt'/>
</auth>
<source protocol='rbd' name='cephstorage/testtrim_system'>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
</source>
<target dev='sda' bus='scsi'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
The ceph docs (https://docs.ceph.com/docs/octopus/rbd/qemu-rbd/) gave me
some hints about enabling trim/discard and I tested using 4M as discard
granularity, but I got the same error resulting in a corrupted ext4 file
system.
Changes made to the libvirt config:
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.scsi0-0-0-0.discard_granularity=4194304'/>
</qemu:commandline>
As the RBD devices are thin-provisioned we really need calling fstrim
inside the VM regularly to free up unused blocks, otherwise our Ceph
pool will run out of space.
Any ideas what could be wrong with our RBD setup or can somebody else
reproduce the problem?
Any hints on how to debug this problem?
Any related/open Ceph issues? (I could not fined one)
I haven't heard of any similar issue. I would recommend trying an
older release of librbd1 (i.e. Nautilus), older release of QEMU, or
older guest OS in different combinations to see what the common factor
is.
Thanks a lot for your help, Georg
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
4 Aug
4 Aug
3:12 a.m.
On 03.08.20 14:56, Jason Dillaman wrote:
On Mon, Aug 3, 2020 at 4:11 AM Georg Schönberger
<g.schoenberger(a)xortex.com> wrote:
Digging deeper into the problem I can tell that fstrim is not the cause
but the trigger that leads to a read-only ext4 filesystem.
Our problem only occurs with VMs that were recently migrated from one
Ceph Cluster to another one with rbd export and import. This is how this
was done:
1. rbd first snap of running VM
2. rbd export-diff of first snap -> rbd import-diff in new cluster
3. Stop VM
4. rbd second snap of stopped VM
5. rbd export-diff of second snap with --from-snap first snap -> rbd
import-diff in new cluster
In some cases we now got corrupted ext4 filesystem with this type of
migration but it is not clear to us why because the VM got stopped in
step 3!
Anything wrong with our sequence of commands?
One thing we think it could a possible cause is the enabled rbd cache in
our ceph.conf:
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = false
rbd_cache_size = 536870912
rbd_cache_max_dirty = 134217728
rbd_cache_target_dirty = 33554432
rbd_cache_max_dirty_age = 5
So if step 4 is directly done after step 3 without waiting for the rbd
cache to be flushed, can this be the cause for data corruption?
Any rbd commands to tell Ceph to flush the rbd cache?
THX, Georg
Hey Ceph users,
we are currently facing some serious problems on our Ceph Cluster with
libvirt (KVM), RBD devices and FSTRIM running inside VMs.
The problem is right after running the fstrim command inside the VM the
ext4 filesystem is corrupted and read-only with the following error message:
EXT4-fs error (device sda1): ext4_mb_generate_buddy:756: group 136,
block bitmap and bg descriptor inconsistent: 32200 vs 32768 free clusters
Aborting journal on device sda1-8
EXT4-fs (sda1): Remounting filesystem read-only
EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected
aborted journal
EXT4-fs (sda1): Remounting filesystem read-only
This behavior is reproducible across several VMs with different OS
(Ubuntu 14.04, 16.04 and 18.04) so we guess it is a bug or a
configuration problem regarding RBD devices.
Our setup on the hosts running the VMs looks like:
# lsb_release -d
Description: Ubuntu 20.04 LTS
# uname -a
Linux XXX 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
# ceph --version
ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus
(stable)
-> I know there's the update to Ceph 15.2.4 but I haven't seen any
fstrim/discard related changes in the changelog. If we could fix the
problem with 15.2.4 I would be happy...
The libvirt config for the RBD device with supporting fstrim (discard)
is the following:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='directsync'
io='native'
discard='unmap'/>
<auth username='libvirt'>
<secret type='ceph' usage='client.libvirt'/>
</auth>
<source protocol='rbd' name='cephstorage/testtrim_system'>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
</source>
<target dev='sda' bus='scsi'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
The ceph docs (https://docs.ceph.com/docs/octopus/rbd/qemu-rbd/) gave me
some hints about enabling trim/discard and I tested using 4M as discard
granularity, but I got the same error resulting in a corrupted ext4 file
system.
Changes made to the libvirt config:
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.scsi0-0-0-0.discard_granularity=4194304'/>
</qemu:commandline>
As the RBD devices are thin-provisioned we really need calling fstrim
inside the VM regularly to free up unused blocks, otherwise our Ceph
pool will run out of space.
Any ideas what could be wrong with our RBD setup or can somebody else
reproduce the problem?
Any hints on how to debug this problem?
Any related/open Ceph issues? (I could not fined one)
I haven't heard of any
similar issue. I would recommend trying an
older release of librbd1 (i.e. Nautilus), older release of QEMU, or
older guest OS in different combinations to see what the common factor
is.
> Thanks a lot for your help, Georg
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
9:19 a.m.
On Tue, Aug 4, 2020 at 2:12 AM Georg Schönberger
<g.schoenberger(a)xortex.com> wrote:
On 03.08.20 14:56, Jason Dillaman wrote:
Ah, well that might be this issue [1]. Can you re-test using the
latest available Octopus dev release of librbd1 / ceph-common (for the
'rbd' CLI) [2]?
On Mon, Aug 3, 2020 at 4:11 AM Georg Schönberger
<g.schoenberger(a)xortex.com> wrote:
Digging deeper into the problem I can tell that fstrim is not the cause
but the trigger that leads to a read-only ext4 filesystem.
Our problem only occurs with VMs that were recently migrated from one
Ceph Cluster to another one with rbd export and import. This is how this
was done:
1. rbd first snap of running VM
2. rbd export-diff of first snap -> rbd import-diff in new cluster
3. Stop VM
4. rbd second snap of stopped VM
5. rbd export-diff of second snap with --from-snap first snap -> rbd
import-diff in new cluster Hey Ceph users,
we are currently facing some serious problems on our Ceph Cluster with
libvirt (KVM), RBD devices and FSTRIM running inside VMs.
The problem is right after running the fstrim command inside the VM the
ext4 filesystem is corrupted and read-only with the following error message:
EXT4-fs error (device sda1): ext4_mb_generate_buddy:756: group 136,
block bitmap and bg descriptor inconsistent: 32200 vs 32768 free clusters
Aborting journal on device sda1-8
EXT4-fs (sda1): Remounting filesystem read-only
EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected
aborted journal
EXT4-fs (sda1): Remounting filesystem read-only
This behavior is reproducible across several VMs with different OS
(Ubuntu 14.04, 16.04 and 18.04) so we guess it is a bug or a
configuration problem regarding RBD devices.
Our setup on the hosts running the VMs looks like:
# lsb_release -d
Description: Ubuntu 20.04 LTS
# uname -a
Linux XXX 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
# ceph --version
ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus
(stable)
-> I know there's the update to Ceph 15.2.4 but I haven't seen any
fstrim/discard related changes in the changelog. If we could fix the
problem with 15.2.4 I would be happy...
The libvirt config for the RBD device with supporting fstrim (discard)
is the following:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='directsync'
io='native'
discard='unmap'/>
<auth username='libvirt'>
<secret type='ceph' usage='client.libvirt'/>
</auth>
<source protocol='rbd' name='cephstorage/testtrim_system'>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
</source>
<target dev='sda' bus='scsi'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
The ceph docs (https://docs.ceph.com/docs/octopus/rbd/qemu-rbd/) gave me
some hints about enabling trim/discard and I tested using 4M as discard
granularity, but I got the same error resulting in a corrupted ext4 file
system.
Changes made to the libvirt config:
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.scsi0-0-0-0.discard_granularity=4194304'/>
</qemu:commandline>
As the RBD devices are thin-provisioned we really need calling fstrim
inside the VM regularly to free up unused blocks, otherwise our Ceph
pool will run out of space.
Any ideas what could be wrong with our RBD setup or can somebody else
reproduce the problem?
Any hints on how to debug this problem?
Any related/open Ceph issues? (I could not fined one)
I haven't heard of any
similar issue. I would recommend trying an
older release of librbd1 (i.e. Nautilus), older release of QEMU, or
older guest OS in different combinations to see what the common factor
is.
> Thanks a lot for your help, Georg
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> In some cases we now got corrupted ext4 filesystem
with this type of
migration but it is not clear to us why because the VM got stopped in
step 3!
Anything wrong with our sequence of commands?
One thing we think it could a possible cause is the enabled rbd cache in
our ceph.conf:
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = false
rbd_cache_size = 536870912
rbd_cache_max_dirty = 134217728
rbd_cache_target_dirty = 33554432
rbd_cache_max_dirty_age = 5
So if step 4 is directly done after step 3 without waiting for the rbd
cache to be flushed, can this be the cause for data corruption?
Any rbd commands to tell Ceph to flush the rbd cache?
THX, Georg
[1] https://tracker.ceph.com/issues/46674
[2] https://shaman.ceph.com/repos/ceph/octopus/
--
Jason
5 Aug
5 Aug
3:17 a.m.
On 04.08.20 14:19, Jason Dillaman wrote:
On Tue, Aug 4, 2020 at 2:12 AM Georg Schönberger
<g.schoenberger(a)xortex.com> wrote:
Issue [1] looks promising. But I might be afraid testing a devel
realease on our productive cluster.
Any chance to test the rbd import otherwise?
On 03.08.20 14:56, Jason Dillaman wrote:
Ah, well that might be this issue [1]. Can you re-test
using the
latest available Octopus dev release of librbd1 / ceph-common (for the
'rbd' CLI) [2]?
On Mon, Aug 3, 2020 at 4:11 AM Georg Schönberger
<g.schoenberger(a)xortex.com> wrote:
Digging deeper into the problem I can tell that fstrim is not the cause
but the trigger that leads to a read-only ext4 filesystem.
Our problem only occurs with VMs that were recently migrated from one
Ceph Cluster to another one with rbd export and import. This is how this
was done:
1. rbd first snap of running VM
2. rbd export-diff of first snap -> rbd import-diff in new cluster
3. Stop VM
4. rbd second snap of stopped VM
5. rbd export-diff of second snap with --from-snap first snap -> rbd
import-diff in new cluster Hey Ceph users,
we are currently facing some serious problems on our Ceph Cluster with
libvirt (KVM), RBD devices and FSTRIM running inside VMs.
The problem is right after running the fstrim command inside the VM the
ext4 filesystem is corrupted and read-only with the following error message:
EXT4-fs error (device sda1): ext4_mb_generate_buddy:756: group 136,
block bitmap and bg descriptor inconsistent: 32200 vs 32768 free clusters
Aborting journal on device sda1-8
EXT4-fs (sda1): Remounting filesystem read-only
EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected
aborted journal
EXT4-fs (sda1): Remounting filesystem read-only
This behavior is reproducible across several VMs with different OS
(Ubuntu 14.04, 16.04 and 18.04) so we guess it is a bug or a
configuration problem regarding RBD devices.
Our setup on the hosts running the VMs looks like:
# lsb_release -d
Description: Ubuntu 20.04 LTS
# uname -a
Linux XXX 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
# ceph --version
ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus
(stable)
-> I know there's the update to Ceph 15.2.4 but I haven't seen any
fstrim/discard related changes in the changelog. If we could fix the
problem with 15.2.4 I would be happy...
The libvirt config for the RBD device with supporting fstrim (discard)
is the following:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='directsync'
io='native'
discard='unmap'/>
<auth username='libvirt'>
<secret type='ceph' usage='client.libvirt'/>
</auth>
<source protocol='rbd'
name='cephstorage/testtrim_system'>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
<host name='XXX' port='6789'/>
</source>
<target dev='sda' bus='scsi'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
The ceph docs (https://docs.ceph.com/docs/octopus/rbd/qemu-rbd/) gave me
some hints about enabling trim/discard and I tested using 4M as discard
granularity, but I got the same error resulting in a corrupted ext4 file
system.
Changes made to the libvirt config:
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.scsi0-0-0-0.discard_granularity=4194304'/>
</qemu:commandline>
As the RBD devices are thin-provisioned we really need calling fstrim
inside the VM regularly to free up unused blocks, otherwise our Ceph
pool will run out of space.
Any ideas what could be wrong with our RBD setup or can somebody else
reproduce the problem?
Any hints on how to debug this problem?
Any related/open Ceph issues? (I could not fined one)
I haven't heard of any
similar issue. I would recommend trying an
older release of librbd1 (i.e. Nautilus), older release of QEMU, or
older guest OS in different combinations to see what the common factor
is.
> Thanks a lot for your help, Georg
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> In some cases we now got corrupted ext4
filesystem with this type of
migration but it is not clear to us why because the VM got stopped in
step 3!
Anything wrong with our sequence of commands?
One thing we think it could a possible cause is the enabled rbd cache in
our ceph.conf:
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = false
rbd_cache_size = 536870912
rbd_cache_max_dirty = 134217728
rbd_cache_target_dirty = 33554432
rbd_cache_max_dirty_age = 5
So if step 4 is directly done after step 3 without waiting for the rbd
cache to be flushed, can this be the cause for data corruption?
Any rbd commands to tell Ceph to flush the rbd cache?
THX, Georg
[1] https://tracker.ceph.com/issues/46674
[2] https://shaman.ceph.com/repos/ceph/octopus/
1371
days inactive
1373
days old
4 comments
2 participants
participants (2)
-
Georg Schönberger -
Jason Dillaman