Help needed please ! Filesystem became read-only !
Hello, First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode". The logs are showing : Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly... After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to. Please find below additional details if needed. Thanks a lot ! Nicolas --- # ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...] # ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB # ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d) data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean
Hi Nicolas, This is a known issue and Venky is working on it, please see https://tracker.ceph.com/issues/63259. Thanks - Xiubo On 6/3/24 20:04, nbarbier@deltaonline.net wrote:
Hello,
First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode".
The logs are showing :
Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly...
After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to.
Please find below additional details if needed. Thanks a lot !
Nicolas
---
# ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...]
# ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB
# ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only
services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d)
data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, A little break into this thread, but I have some questions: * How does this happen, that the filesystem gets into readonly modus * Is this avoidable? * How-to fix the issue, because I didn't see a workaround in the mentioned tracker (or I missed it) * With this bug around, should you use cephfs with reef? Kind regards, Sake
Op 04-06-2024 04:04 CEST schreef Xiubo Li <xiubli@redhat.com>:
Hi Nicolas,
This is a known issue and Venky is working on it, please see https://tracker.ceph.com/issues/63259.
Thanks - Xiubo
On 6/3/24 20:04, nbarbier@deltaonline.net wrote:
Hello,
First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode".
The logs are showing :
Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly...
After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to.
Please find below additional details if needed. Thanks a lot !
Nicolas
---
# ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...]
# ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB
# ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only
services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d)
data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
First, thanks Xiubo for your feedback ! To go further on the points raised by Sake: - How does this happen ? -> There were no preliminary signs before the incident - Is this avoidable? -> Good question, I'd also like to know how! - How to fix the issue ? -> So far, no fix nor workaround from what I read. I am very interested in finding a way to have the storage running again, so far our cluster is out of order as it's not possible to write on it anymore (good to know that data is still readable btw!). I'm not a Ceph guru so I don't want to play with the settings / parameters as the result could be even worse, but help would be greatly appreciated to get a system back available ! - Should you use cephfs with reef ? -> Well, from my experience, not for production Thanks to everyone who helped me or will help me find a solution! Nicolas
Hi, I believe our KL studio has hit this same bug after deleting a pool that was used only for testing. So, is there any procedure to get rid of those bad journal events and get the mds back to rw state? Thanks, --------------------------- Olli Rajala - Lead TD Anima Vitae Ltd. www.anima.fi --------------------------- On Wed, Jun 5, 2024 at 4:57 AM <nbarbier@deltaonline.net> wrote:
First, thanks Xiubo for your feedback !
To go further on the points raised by Sake: - How does this happen ? -> There were no preliminary signs before the incident
- Is this avoidable? -> Good question, I'd also like to know how!
- How to fix the issue ? -> So far, no fix nor workaround from what I read. I am very interested in finding a way to have the storage running again, so far our cluster is out of order as it's not possible to write on it anymore (good to know that data is still readable btw!). I'm not a Ceph guru so I don't want to play with the settings / parameters as the result could be even worse, but help would be greatly appreciated to get a system back available !
- Should you use cephfs with reef ? -> Well, from my experience, not for production
Thanks to everyone who helped me or will help me find a solution! Nicolas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/4/24 15:20, Sake Ceph wrote:
Hi,
A little break into this thread, but I have some questions: * How does this happen, that the filesystem gets into readonly modus
The detail explanation you can refer to the ceph PR: https://github.com/ceph/ceph/pull/55421.
* Is this avoidable? * How-to fix the issue, because I didn't see a workaround in the mentioned tracker (or I missed it) Possibly avoid changing data pools or disable multiple data pools? * With this bug around, should you use cephfs with reef?
This will happen in all the releases, so that doesn't matter. - Xiubo
Kind regards, Sake
Op 04-06-2024 04:04 CEST schreef Xiubo Li <xiubli@redhat.com>:
Hi Nicolas,
This is a known issue and Venky is working on it, please see https://tracker.ceph.com/issues/63259.
Thanks - Xiubo
On 6/3/24 20:04, nbarbier@deltaonline.net wrote:
Hello,
First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode".
The logs are showing :
Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly...
After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to.
Please find below additional details if needed. Thanks a lot !
Nicolas
---
# ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...]
# ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB
# ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only
services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d)
data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Xiubo Thank you for the explanation! This won't be a issue for us, but made me think twice :) Kind regards, Sake
Op 04-06-2024 12:30 CEST schreef Xiubo Li <xiubli@redhat.com>:
On 6/4/24 15:20, Sake Ceph wrote:
Hi,
A little break into this thread, but I have some questions: * How does this happen, that the filesystem gets into readonly modus
The detail explanation you can refer to the ceph PR: https://github.com/ceph/ceph/pull/55421.
* Is this avoidable? * How-to fix the issue, because I didn't see a workaround in the mentioned tracker (or I missed it) Possibly avoid changing data pools or disable multiple data pools? * With this bug around, should you use cephfs with reef?
This will happen in all the releases, so that doesn't matter.
- Xiubo
Kind regards, Sake
Op 04-06-2024 04:04 CEST schreef Xiubo Li <xiubli@redhat.com>:
Hi Nicolas,
This is a known issue and Venky is working on it, please see https://tracker.ceph.com/issues/63259.
Thanks - Xiubo
On 6/3/24 20:04, nbarbier@deltaonline.net wrote:
Hello,
First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode".
The logs are showing :
Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly...
After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to.
Please find below additional details if needed. Thanks a lot !
Nicolas
---
# ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...]
# ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB
# ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only
services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d)
data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
nbarbier＠deltaonline.net
-
Olli Rajala
-
Sake Ceph
-
Xiubo Li