On 6/4/24 15:20, Sake Ceph wrote:
Hi,
A little break into this thread, but I have some questions: * How does this happen, that the filesystem gets into readonly modus
The detail explanation you can refer to the ceph PR: https://github.com/ceph/ceph/pull/55421.
* Is this avoidable? * How-to fix the issue, because I didn't see a workaround in the mentioned tracker (or I missed it) Possibly avoid changing data pools or disable multiple data pools? * With this bug around, should you use cephfs with reef?
This will happen in all the releases, so that doesn't matter. - Xiubo
Kind regards, Sake
Op 04-06-2024 04:04 CEST schreef Xiubo Li <xiubli@redhat.com>:
Hi Nicolas,
This is a known issue and Venky is working on it, please see https://tracker.ceph.com/issues/63259.
Thanks - Xiubo
On 6/3/24 20:04, nbarbier@deltaonline.net wrote:
Hello,
First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode".
The logs are showing :
Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly...
After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to.
Please find below additional details if needed. Thanks a lot !
Nicolas
---
# ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...]
# ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata
# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB
# ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only
services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d)
data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io