MDS in ReadOnly and 2 MDS behind on trimming
Dear Ceph Community, I am having an issue with my Ceph Cluster , there were several osd crashing but now active and recovery finished and now the CephFS filesystem cannot be access by clients in RW (K8S worklod) as the 1 MDS is in Read-Only and 2 are being on trimming The cephfs seems to have volume OK The trimming process seems not going further, maybe stuck ? We are running 3 hosts using ceph Pacific version 16.2.1 Here some logs on the situation : ceph versions { "mon": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3 }, "osd": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 18 }, "mds": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3 }, "rgw": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 6 }, "overall": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 33 } } ceph orch ps NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID crash.rke-sh1-1 rke-sh1-1 running (21h) 36s ago 21h - 16.2.1 c757e4a3636b e8652edb2b49 crash.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 20M - 16.2.1 c757e4a3636b a1249a605ee0 crash.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b 026667bc1776 mds.cephfs.rke-sh1-1.ojmpnk rke-sh1-1 running (18h) 36s ago 4M - 16.2.1 c757e4a3636b 9b4c2b08b759 mds.cephfs.rke-sh1-2.isqjza rke-sh1-2 running (18h) 3m ago 23M - 16.2.1 c757e4a3636b 71681a5f34d3 mds.cephfs.rke-sh1-3.vdicdn rke-sh1-3 running (17h) 36s ago 3M - 16.2.1 c757e4a3636b e89946ad6b7e mgr.rke-sh1-1.qskoyj rke-sh1-1 running (21h) 36s ago 2y *:8082 *:9283 16.2.1 c757e4a3636b 7ce7cfbb3e55 mgr.rke-sh1-2.lxmguj rke-sh1-2 running (21h) 3m ago 22M *:8082 *:9283 16.2.1 c757e4a3636b 5a0025adfd46 mgr.rke-sh1-3.ckunvo rke-sh1-3 running (17h) 36s ago 6M *:8082 *:9283 16.2.1 c757e4a3636b 2fcaf18f3218 mon.rke-sh1-1 rke-sh1-1 running (20h) 36s ago 20h - 16.2.1 c757e4a3636b c0a90103cabc mon.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 3M - 16.2.1 c757e4a3636b f4b32ba4466b mon.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b d5e44c245998 osd.0 rke-sh1-2 running (20h) 3m ago 2y - 16.2.1 c757e4a3636b 7b0e69942c15 osd.1 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 4451654d9a2d osd.10 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 3f9d5f95e284 osd.11 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b db1cc6d2e37f osd.12 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b de416c1ef766 osd.13 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 25a281cc5a9b osd.14 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 62f25ba61667 osd.15 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b d3514d823c45 osd.16 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b bba857759bfe osd.17 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 59281d4bb3d0 osd.2 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 418041b5e60d osd.3 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 04a0e29d5623 osd.4 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b 1cc78a5153d3 osd.5 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 39a4b11e31fb osd.6 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 2f218ffb566e osd.7 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b cf761fbe4d5f osd.8 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b f9f85480e800 osd.9 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 664c54ff46d2 rgw.default.rke-sh1-1.dgucwl rke-sh1-1 running (21h) 36s ago 22M *:8000 16.2.1 c757e4a3636b f03212b955a7 rgw.default.rke-sh1-1.vylchc rke-sh1-1 running (21h) 36s ago 22M *:8001 16.2.1 c757e4a3636b da486ce43fe5 rgw.default.rke-sh1-2.dfhhfw rke-sh1-2 running (21h) 3m ago 2y *:8000 16.2.1 c757e4a3636b ef4089d0aef2 rgw.default.rke-sh1-2.efkbum rke-sh1-2 running (21h) 3m ago 2y *:8001 16.2.1 c757e4a3636b 9e053d5a2f7b rgw.default.rke-sh1-3.krfgey rke-sh1-3 running (17h) 36s ago 9M *:8001 16.2.1 c757e4a3636b 45cd3d75edd3 rgw.default.rke-sh1-3.pwdbmp rke-sh1-3 running (17h) 36s ago 9M *:8000 16.2.1 c757e4a3636b e2710265a7f4 ceph health detail HEALTH_WARN 1 MDSs are read only; 2 MDSs behind on trimming [WRN] MDS_READ_ONLY: 1 MDSs are read only mds.cephfs.rke-sh1-2.isqjza(mds.0): MDS in read-only mode [WRN] MDS_TRIM: 2 MDSs behind on trimming mds.cephfs.rke-sh1-2.isqjza(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149 mds.cephfs.rke-sh1-1.ojmpnk(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149 root@rke-sh1-1:~# ceph fs status cephfs - 27 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.rke-sh1-2.isqjza Reqs: 8 /s 85.2k 53.2k 1742 101 0-s standby-replay cephfs.rke-sh1-1.ojmpnk Evts: 0 /s 52.2k 20.2k 1737 0 POOL TYPE USED AVAIL cephfs_metadata metadata 1109G 6082G cephfs_data data 8419G 6082G STANDBY MDS cephfs.rke-sh1-3.vdicdn MDS version: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable) ceph status cluster: id: fcb373ce-7aaa-11eb-984f-e7c6e0038e87 health: HEALTH_WARN 1 MDSs are read only 2 MDSs behind on trimming services: mon: 3 daemons, quorum rke-sh1-2,rke-sh1-1,rke-sh1-3 (age 17h) mgr: rke-sh1-1.qskoyj(active, since 17h), standbys: rke-sh1-2.lxmguj, rke-sh1-3.ckunvo mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 18 osds: 18 up (since 17h), 18 in (since 20h) rgw: 6 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 849 pgs objects: 10.10M objects, 5.3 TiB usage: 11 TiB used, 15 TiB / 26 TiB avail pgs: 849 active+clean io: client: 35 KiB/s rd, 1.0 MiB/s wr, 302 op/s rd, 165 op/s wr # ceph mds stat cephfs:1 {0=cephfs.rke-sh1-2.isqjza=up:active} 1 up:standby-replay 1 up:standby Have you got an idea on what could be my next steps to bring the cluster healthy ? Help will very be appreciated. Thank a lot for your feedback. Best Regards, Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 <https://www.csti.ch/> www.csti.ch
Hi, the mds log should contain information why it goes into read-only mode. Just a few weeks ago I helped a user with a broken CephFS (MDS went into read-only mode because of missing objects in the journal). Can you check the journal status: # cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect # cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect and also share the logs. Thanks, Eugen Zitat von Edouard FAZENDA <e.fazenda@csti.ch>:
Dear Ceph Community,
I am having an issue with my Ceph Cluster , there were several osd crashing but now active and recovery finished and now the CephFS filesystem cannot be access by clients in RW (K8S worklod) as the 1 MDS is in Read-Only and 2 are being on trimming
The cephfs seems to have volume OK
The trimming process seems not going further, maybe stuck ?
We are running 3 hosts using ceph Pacific version 16.2.1
Here some logs on the situation :
ceph versions
{
"mon": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"mgr": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"osd": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 18
},
"mds": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"rgw": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 6
},
"overall": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 33
}
}
ceph orch ps
NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID
crash.rke-sh1-1 rke-sh1-1 running (21h) 36s ago 21h - 16.2.1 c757e4a3636b e8652edb2b49
crash.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 20M - 16.2.1 c757e4a3636b a1249a605ee0
crash.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b 026667bc1776
mds.cephfs.rke-sh1-1.ojmpnk rke-sh1-1 running (18h) 36s ago 4M - 16.2.1 c757e4a3636b 9b4c2b08b759
mds.cephfs.rke-sh1-2.isqjza rke-sh1-2 running (18h) 3m ago 23M - 16.2.1 c757e4a3636b 71681a5f34d3
mds.cephfs.rke-sh1-3.vdicdn rke-sh1-3 running (17h) 36s ago 3M - 16.2.1 c757e4a3636b e89946ad6b7e
mgr.rke-sh1-1.qskoyj rke-sh1-1 running (21h) 36s ago 2y *:8082 *:9283 16.2.1 c757e4a3636b 7ce7cfbb3e55
mgr.rke-sh1-2.lxmguj rke-sh1-2 running (21h) 3m ago 22M *:8082 *:9283 16.2.1 c757e4a3636b 5a0025adfd46
mgr.rke-sh1-3.ckunvo rke-sh1-3 running (17h) 36s ago 6M *:8082 *:9283 16.2.1 c757e4a3636b 2fcaf18f3218
mon.rke-sh1-1 rke-sh1-1 running (20h) 36s ago 20h - 16.2.1 c757e4a3636b c0a90103cabc
mon.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 3M - 16.2.1 c757e4a3636b f4b32ba4466b
mon.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b d5e44c245998
osd.0 rke-sh1-2 running (20h) 3m ago 2y - 16.2.1 c757e4a3636b 7b0e69942c15
osd.1 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 4451654d9a2d
osd.10 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 3f9d5f95e284
osd.11 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b db1cc6d2e37f
osd.12 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b de416c1ef766
osd.13 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 25a281cc5a9b
osd.14 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 62f25ba61667
osd.15 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b d3514d823c45
osd.16 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b bba857759bfe
osd.17 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 59281d4bb3d0
osd.2 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 418041b5e60d
osd.3 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 04a0e29d5623
osd.4 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b 1cc78a5153d3
osd.5 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 39a4b11e31fb
osd.6 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 2f218ffb566e
osd.7 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b cf761fbe4d5f
osd.8 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b f9f85480e800
osd.9 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 664c54ff46d2
rgw.default.rke-sh1-1.dgucwl rke-sh1-1 running (21h) 36s ago 22M *:8000 16.2.1 c757e4a3636b f03212b955a7
rgw.default.rke-sh1-1.vylchc rke-sh1-1 running (21h) 36s ago 22M *:8001 16.2.1 c757e4a3636b da486ce43fe5
rgw.default.rke-sh1-2.dfhhfw rke-sh1-2 running (21h) 3m ago 2y *:8000 16.2.1 c757e4a3636b ef4089d0aef2
rgw.default.rke-sh1-2.efkbum rke-sh1-2 running (21h) 3m ago 2y *:8001 16.2.1 c757e4a3636b 9e053d5a2f7b
rgw.default.rke-sh1-3.krfgey rke-sh1-3 running (17h) 36s ago 9M *:8001 16.2.1 c757e4a3636b 45cd3d75edd3
rgw.default.rke-sh1-3.pwdbmp rke-sh1-3 running (17h) 36s ago 9M *:8000 16.2.1 c757e4a3636b e2710265a7f4
ceph health detail
HEALTH_WARN 1 MDSs are read only; 2 MDSs behind on trimming
[WRN] MDS_READ_ONLY: 1 MDSs are read only
mds.cephfs.rke-sh1-2.isqjza(mds.0): MDS in read-only mode
[WRN] MDS_TRIM: 2 MDSs behind on trimming
mds.cephfs.rke-sh1-2.isqjza(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149
mds.cephfs.rke-sh1-1.ojmpnk(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149
root@rke-sh1-1:~# ceph fs status
cephfs - 27 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.rke-sh1-2.isqjza Reqs: 8 /s 85.2k 53.2k 1742 101
0-s standby-replay cephfs.rke-sh1-1.ojmpnk Evts: 0 /s 52.2k 20.2k 1737 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 1109G 6082G
cephfs_data data 8419G 6082G
STANDBY MDS
cephfs.rke-sh1-3.vdicdn
MDS version: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
ceph status
cluster:
id: fcb373ce-7aaa-11eb-984f-e7c6e0038e87
health: HEALTH_WARN
1 MDSs are read only
2 MDSs behind on trimming
services:
mon: 3 daemons, quorum rke-sh1-2,rke-sh1-1,rke-sh1-3 (age 17h)
mgr: rke-sh1-1.qskoyj(active, since 17h), standbys: rke-sh1-2.lxmguj, rke-sh1-3.ckunvo
mds: 1/1 daemons up, 1 standby, 1 hot standby
osd: 18 osds: 18 up (since 17h), 18 in (since 20h)
rgw: 6 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 849 pgs
objects: 10.10M objects, 5.3 TiB
usage: 11 TiB used, 15 TiB / 26 TiB avail
pgs: 849 active+clean
io:
client: 35 KiB/s rd, 1.0 MiB/s wr, 302 op/s rd, 165 op/s wr
# ceph mds stat
cephfs:1 {0=cephfs.rke-sh1-2.isqjza=up:active} 1 up:standby-replay 1 up:standby
Have you got an idea on what could be my next steps to bring the cluster healthy ?
Help will very be appreciated.
Thank a lot for your feedback.
Best Regards,
Edouard FAZENDA
Technical Support
Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40
<https://www.csti.ch/> www.csti.ch
Hi Eugen, Thanks for the reply, really appreciate The first command , just hang with no output # cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect The second command # cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect Overall journal integrity: OK root@rke-sh1-2:~# cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mds.cephfs.rke-sh1-2.isqjza -- Logs begin at Fri 2024-02-23 04:49:32 UTC, end at Fri 2024-02-23 13:08:22 UTC. -- Feb 23 07:46:46 rke-sh1-2 bash[1058012]: ignoring --setuser ceph since I am not root Feb 23 07:46:46 rke-sh1-2 bash[1058012]: ignoring --setgroup ceph since I am not root Feb 23 07:46:46 rke-sh1-2 bash[1058012]: starting mds.cephfs.rke-sh1-2.isqjza at Feb 23 08:15:06 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:06.371+0000 7fbc17dd9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. Feb 23 08:15:13 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x1 object, errno -22 Feb 23 08:15:13 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 mds.0.12487 unhandled write error (22) Invalid argument, force readonly... Feb 23 10:20:36 rke-sh1-2 bash[1058012]: debug 2024-02-23T10:20:36.309+0000 7fbc17dd9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. root@rke-sh1-3:~# cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mds.cephfs.rke-sh1-3.vdicdn -- Logs begin at Fri 2024-02-23 06:59:48 UTC, end at Fri 2024-02-23 13:09:18 UTC. -- Feb 23 07:46:46 rke-sh1-3 bash[2901]: ignoring --setuser ceph since I am not root Feb 23 07:46:46 rke-sh1-3 bash[2901]: ignoring --setgroup ceph since I am not root Feb 23 07:46:46 rke-sh1-3 bash[2901]: starting mds.cephfs.rke-sh1-3.vdicdn at Feb 23 10:25:51 rke-sh1-3 bash[2901]: ignoring --setuser ceph since I am not root Feb 23 10:25:51 rke-sh1-3 bash[2901]: ignoring --setgroup ceph since I am not root Feb 23 10:25:51 rke-sh1-3 bash[2901]: starting mds.cephfs.rke-sh1-3.vdicdn at debug2: channel 0: request window-change confirm 0 debug3: send packet: type 98 -- Logs begin at Fri 2024-02-23 00:24:42 UTC, end at Fri 2024-02-23 13:09:55 UTC. -- Feb 23 09:29:10 rke-sh1-1 bash[786820]: tcmalloc: large alloc 1073750016 bytes == 0x5598512de000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41da6> Feb 23 09:29:19 rke-sh1-1 bash[786820]: tcmalloc: large alloc 2147491840 bytes == 0x559891ae0000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41db3> Feb 23 09:29:26 rke-sh1-1 bash[786820]: tcmalloc: large alloc 2147491840 bytes == 0x559951ae4000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41da6> Feb 23 09:29:27 rke-sh1-1 bash[786820]: debug 2024-02-23T09:29:27.928+0000 7fb416d63700 -1 asok(0x5597c3904000) AdminSocket: error writing response length (32) Broken pipe Feb 23 12:35:53 rke-sh1-1 bash[786820]: ignoring --setuser ceph since I am not root Feb 23 12:35:53 rke-sh1-1 bash[786820]: ignoring --setgroup ceph since I am not root Feb 23 12:35:53 rke-sh1-1 bash[786820]: starting mds.cephfs.rke-sh1-1.ojmpnk at The logs of the MDS are in verbose 20 , do you want me to provide on a archive ? Is there a way to compact all the logs ? Best Regards, Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 www.csti.ch -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: vendredi, 23 février 2024 12:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: MDS in ReadOnly and 2 MDS behind on trimming Hi, the mds log should contain information why it goes into read-only mode. Just a few weeks ago I helped a user with a broken CephFS (MDS went into read-only mode because of missing objects in the journal). Can you check the journal status: # cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect # cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect and also share the logs. Thanks, Eugen Zitat von Edouard FAZENDA <e.fazenda@csti.ch>:
Dear Ceph Community,
I am having an issue with my Ceph Cluster , there were several osd crashing but now active and recovery finished and now the CephFS filesystem cannot be access by clients in RW (K8S worklod) as the 1 MDS is in Read-Only and 2 are being on trimming
The cephfs seems to have volume OK
The trimming process seems not going further, maybe stuck ?
We are running 3 hosts using ceph Pacific version 16.2.1
Here some logs on the situation :
ceph versions
{
"mon": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"mgr": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"osd": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 18
},
"mds": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"rgw": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 6
},
"overall": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 33
}
}
ceph orch ps
NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID
crash.rke-sh1-1 rke-sh1-1 running (21h) 36s ago 21h - 16.2.1 c757e4a3636b e8652edb2b49
crash.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 20M - 16.2.1 c757e4a3636b a1249a605ee0
crash.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b 026667bc1776
mds.cephfs.rke-sh1-1.ojmpnk rke-sh1-1 running (18h) 36s ago 4M - 16.2.1 c757e4a3636b 9b4c2b08b759
mds.cephfs.rke-sh1-2.isqjza rke-sh1-2 running (18h) 3m ago 23M - 16.2.1 c757e4a3636b 71681a5f34d3
mds.cephfs.rke-sh1-3.vdicdn rke-sh1-3 running (17h) 36s ago 3M - 16.2.1 c757e4a3636b e89946ad6b7e
mgr.rke-sh1-1.qskoyj rke-sh1-1 running (21h) 36s ago 2y *:8082 *:9283 16.2.1 c757e4a3636b 7ce7cfbb3e55
mgr.rke-sh1-2.lxmguj rke-sh1-2 running (21h) 3m ago 22M *:8082 *:9283 16.2.1 c757e4a3636b 5a0025adfd46
mgr.rke-sh1-3.ckunvo rke-sh1-3 running (17h) 36s ago 6M *:8082 *:9283 16.2.1 c757e4a3636b 2fcaf18f3218
mon.rke-sh1-1 rke-sh1-1 running (20h) 36s ago 20h - 16.2.1 c757e4a3636b c0a90103cabc
mon.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 3M - 16.2.1 c757e4a3636b f4b32ba4466b
mon.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b d5e44c245998
osd.0 rke-sh1-2 running (20h) 3m ago 2y - 16.2.1 c757e4a3636b 7b0e69942c15
osd.1 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 4451654d9a2d
osd.10 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 3f9d5f95e284
osd.11 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b db1cc6d2e37f
osd.12 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b de416c1ef766
osd.13 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 25a281cc5a9b
osd.14 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 62f25ba61667
osd.15 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b d3514d823c45
osd.16 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b bba857759bfe
osd.17 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 59281d4bb3d0
osd.2 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 418041b5e60d
osd.3 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 04a0e29d5623
osd.4 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b 1cc78a5153d3
osd.5 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 39a4b11e31fb
osd.6 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 2f218ffb566e
osd.7 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b cf761fbe4d5f
osd.8 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b f9f85480e800
osd.9 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 664c54ff46d2
rgw.default.rke-sh1-1.dgucwl rke-sh1-1 running (21h) 36s ago 22M *:8000 16.2.1 c757e4a3636b f03212b955a7
rgw.default.rke-sh1-1.vylchc rke-sh1-1 running (21h) 36s ago 22M *:8001 16.2.1 c757e4a3636b da486ce43fe5
rgw.default.rke-sh1-2.dfhhfw rke-sh1-2 running (21h) 3m ago 2y *:8000 16.2.1 c757e4a3636b ef4089d0aef2
rgw.default.rke-sh1-2.efkbum rke-sh1-2 running (21h) 3m ago 2y *:8001 16.2.1 c757e4a3636b 9e053d5a2f7b
rgw.default.rke-sh1-3.krfgey rke-sh1-3 running (17h) 36s ago 9M *:8001 16.2.1 c757e4a3636b 45cd3d75edd3
rgw.default.rke-sh1-3.pwdbmp rke-sh1-3 running (17h) 36s ago 9M *:8000 16.2.1 c757e4a3636b e2710265a7f4
ceph health detail
HEALTH_WARN 1 MDSs are read only; 2 MDSs behind on trimming
[WRN] MDS_READ_ONLY: 1 MDSs are read only
mds.cephfs.rke-sh1-2.isqjza(mds.0): MDS in read-only mode
[WRN] MDS_TRIM: 2 MDSs behind on trimming
mds.cephfs.rke-sh1-2.isqjza(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149
mds.cephfs.rke-sh1-1.ojmpnk(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149
root@rke-sh1-1:~# ceph fs status
cephfs - 27 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.rke-sh1-2.isqjza Reqs: 8 /s 85.2k 53.2k 1742 101
0-s standby-replay cephfs.rke-sh1-1.ojmpnk Evts: 0 /s 52.2k 20.2k 1737 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 1109G 6082G
cephfs_data data 8419G 6082G
STANDBY MDS
cephfs.rke-sh1-3.vdicdn
MDS version: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
ceph status
cluster:
id: fcb373ce-7aaa-11eb-984f-e7c6e0038e87
health: HEALTH_WARN
1 MDSs are read only
2 MDSs behind on trimming
services:
mon: 3 daemons, quorum rke-sh1-2,rke-sh1-1,rke-sh1-3 (age 17h)
mgr: rke-sh1-1.qskoyj(active, since 17h), standbys: rke-sh1-2.lxmguj, rke-sh1-3.ckunvo
mds: 1/1 daemons up, 1 standby, 1 hot standby
osd: 18 osds: 18 up (since 17h), 18 in (since 20h)
rgw: 6 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 849 pgs
objects: 10.10M objects, 5.3 TiB
usage: 11 TiB used, 15 TiB / 26 TiB avail
pgs: 849 active+clean
io:
client: 35 KiB/s rd, 1.0 MiB/s wr, 302 op/s rd, 165 op/s wr
# ceph mds stat
cephfs:1 {0=cephfs.rke-sh1-2.isqjza=up:active} 1 up:standby-replay 1 up:standby
Have you got an idea on what could be my next steps to bring the cluster healthy ?
Help will very be appreciated.
Thank a lot for your feedback.
Best Regards,
Edouard FAZENDA
Technical Support
Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40
<https://www.csti.ch/> www.csti.ch
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x1 object, errno -22 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 mds.0.12487 unhandled write error (22) Invalid argument, force readonly...
Was your cephfs metadata pool full? This tracker (https://tracker.ceph.com/issues/52260) sounds very similar but I don't see a solution for it. Zitat von Edouard FAZENDA <e.fazenda@csti.ch>:
Hi Eugen,
Thanks for the reply, really appreciate
The first command , just hang with no output # cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect
The second command
# cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect Overall journal integrity: OK
root@rke-sh1-2:~# cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mds.cephfs.rke-sh1-2.isqjza -- Logs begin at Fri 2024-02-23 04:49:32 UTC, end at Fri 2024-02-23 13:08:22 UTC. -- Feb 23 07:46:46 rke-sh1-2 bash[1058012]: ignoring --setuser ceph since I am not root Feb 23 07:46:46 rke-sh1-2 bash[1058012]: ignoring --setgroup ceph since I am not root Feb 23 07:46:46 rke-sh1-2 bash[1058012]: starting mds.cephfs.rke-sh1-2.isqjza at Feb 23 08:15:06 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:06.371+0000 7fbc17dd9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. Feb 23 08:15:13 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x1 object, errno -22 Feb 23 08:15:13 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 mds.0.12487 unhandled write error (22) Invalid argument, force readonly... Feb 23 10:20:36 rke-sh1-2 bash[1058012]: debug 2024-02-23T10:20:36.309+0000 7fbc17dd9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request.
root@rke-sh1-3:~# cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mds.cephfs.rke-sh1-3.vdicdn -- Logs begin at Fri 2024-02-23 06:59:48 UTC, end at Fri 2024-02-23 13:09:18 UTC. -- Feb 23 07:46:46 rke-sh1-3 bash[2901]: ignoring --setuser ceph since I am not root Feb 23 07:46:46 rke-sh1-3 bash[2901]: ignoring --setgroup ceph since I am not root Feb 23 07:46:46 rke-sh1-3 bash[2901]: starting mds.cephfs.rke-sh1-3.vdicdn at Feb 23 10:25:51 rke-sh1-3 bash[2901]: ignoring --setuser ceph since I am not root Feb 23 10:25:51 rke-sh1-3 bash[2901]: ignoring --setgroup ceph since I am not root Feb 23 10:25:51 rke-sh1-3 bash[2901]: starting mds.cephfs.rke-sh1-3.vdicdn at
debug2: channel 0: request window-change confirm 0 debug3: send packet: type 98 -- Logs begin at Fri 2024-02-23 00:24:42 UTC, end at Fri 2024-02-23 13:09:55 UTC. -- Feb 23 09:29:10 rke-sh1-1 bash[786820]: tcmalloc: large alloc 1073750016 bytes == 0x5598512de000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41da6> Feb 23 09:29:19 rke-sh1-1 bash[786820]: tcmalloc: large alloc 2147491840 bytes == 0x559891ae0000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41db3> Feb 23 09:29:26 rke-sh1-1 bash[786820]: tcmalloc: large alloc 2147491840 bytes == 0x559951ae4000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41da6> Feb 23 09:29:27 rke-sh1-1 bash[786820]: debug 2024-02-23T09:29:27.928+0000 7fb416d63700 -1 asok(0x5597c3904000) AdminSocket: error writing response length (32) Broken pipe Feb 23 12:35:53 rke-sh1-1 bash[786820]: ignoring --setuser ceph since I am not root Feb 23 12:35:53 rke-sh1-1 bash[786820]: ignoring --setgroup ceph since I am not root Feb 23 12:35:53 rke-sh1-1 bash[786820]: starting mds.cephfs.rke-sh1-1.ojmpnk at
The logs of the MDS are in verbose 20 , do you want me to provide on a archive ?
Is there a way to compact all the logs ?
Best Regards,
Edouard FAZENDA Technical Support
Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40
www.csti.ch
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: vendredi, 23 février 2024 12:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: MDS in ReadOnly and 2 MDS behind on trimming
Hi,
the mds log should contain information why it goes into read-only mode. Just a few weeks ago I helped a user with a broken CephFS (MDS went into read-only mode because of missing objects in the journal). Can you check the journal status:
# cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect
# cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect
and also share the logs.
Thanks, Eugen
Zitat von Edouard FAZENDA <e.fazenda@csti.ch>:
Dear Ceph Community,
I am having an issue with my Ceph Cluster , there were several osd crashing but now active and recovery finished and now the CephFS filesystem cannot be access by clients in RW (K8S worklod) as the 1 MDS is in Read-Only and 2 are being on trimming
The cephfs seems to have volume OK
The trimming process seems not going further, maybe stuck ?
We are running 3 hosts using ceph Pacific version 16.2.1
Here some logs on the situation :
ceph versions
{
"mon": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"mgr": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"osd": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 18
},
"mds": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"rgw": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 6
},
"overall": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 33
}
}
ceph orch ps
NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID
crash.rke-sh1-1 rke-sh1-1 running (21h) 36s ago 21h - 16.2.1 c757e4a3636b e8652edb2b49
crash.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 20M - 16.2.1 c757e4a3636b a1249a605ee0
crash.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b 026667bc1776
mds.cephfs.rke-sh1-1.ojmpnk rke-sh1-1 running (18h) 36s ago 4M - 16.2.1 c757e4a3636b 9b4c2b08b759
mds.cephfs.rke-sh1-2.isqjza rke-sh1-2 running (18h) 3m ago 23M - 16.2.1 c757e4a3636b 71681a5f34d3
mds.cephfs.rke-sh1-3.vdicdn rke-sh1-3 running (17h) 36s ago 3M - 16.2.1 c757e4a3636b e89946ad6b7e
mgr.rke-sh1-1.qskoyj rke-sh1-1 running (21h) 36s ago 2y *:8082 *:9283 16.2.1 c757e4a3636b 7ce7cfbb3e55
mgr.rke-sh1-2.lxmguj rke-sh1-2 running (21h) 3m ago 22M *:8082 *:9283 16.2.1 c757e4a3636b 5a0025adfd46
mgr.rke-sh1-3.ckunvo rke-sh1-3 running (17h) 36s ago 6M *:8082 *:9283 16.2.1 c757e4a3636b 2fcaf18f3218
mon.rke-sh1-1 rke-sh1-1 running (20h) 36s ago 20h - 16.2.1 c757e4a3636b c0a90103cabc
mon.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 3M - 16.2.1 c757e4a3636b f4b32ba4466b
mon.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b d5e44c245998
osd.0 rke-sh1-2 running (20h) 3m ago 2y - 16.2.1 c757e4a3636b 7b0e69942c15
osd.1 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 4451654d9a2d
osd.10 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 3f9d5f95e284
osd.11 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b db1cc6d2e37f
osd.12 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b de416c1ef766
osd.13 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 25a281cc5a9b
osd.14 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 62f25ba61667
osd.15 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b d3514d823c45
osd.16 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b bba857759bfe
osd.17 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 59281d4bb3d0
osd.2 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 418041b5e60d
osd.3 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 04a0e29d5623
osd.4 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b 1cc78a5153d3
osd.5 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 39a4b11e31fb
osd.6 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 2f218ffb566e
osd.7 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b cf761fbe4d5f
osd.8 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b f9f85480e800
osd.9 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 664c54ff46d2
rgw.default.rke-sh1-1.dgucwl rke-sh1-1 running (21h) 36s ago 22M *:8000 16.2.1 c757e4a3636b f03212b955a7
rgw.default.rke-sh1-1.vylchc rke-sh1-1 running (21h) 36s ago 22M *:8001 16.2.1 c757e4a3636b da486ce43fe5
rgw.default.rke-sh1-2.dfhhfw rke-sh1-2 running (21h) 3m ago 2y *:8000 16.2.1 c757e4a3636b ef4089d0aef2
rgw.default.rke-sh1-2.efkbum rke-sh1-2 running (21h) 3m ago 2y *:8001 16.2.1 c757e4a3636b 9e053d5a2f7b
rgw.default.rke-sh1-3.krfgey rke-sh1-3 running (17h) 36s ago 9M *:8001 16.2.1 c757e4a3636b 45cd3d75edd3
rgw.default.rke-sh1-3.pwdbmp rke-sh1-3 running (17h) 36s ago 9M *:8000 16.2.1 c757e4a3636b e2710265a7f4
ceph health detail
HEALTH_WARN 1 MDSs are read only; 2 MDSs behind on trimming
[WRN] MDS_READ_ONLY: 1 MDSs are read only
mds.cephfs.rke-sh1-2.isqjza(mds.0): MDS in read-only mode
[WRN] MDS_TRIM: 2 MDSs behind on trimming
mds.cephfs.rke-sh1-2.isqjza(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149
mds.cephfs.rke-sh1-1.ojmpnk(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149
root@rke-sh1-1:~# ceph fs status
cephfs - 27 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.rke-sh1-2.isqjza Reqs: 8 /s 85.2k 53.2k 1742 101
0-s standby-replay cephfs.rke-sh1-1.ojmpnk Evts: 0 /s 52.2k 20.2k 1737 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 1109G 6082G
cephfs_data data 8419G 6082G
STANDBY MDS
cephfs.rke-sh1-3.vdicdn
MDS version: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
ceph status
cluster:
id: fcb373ce-7aaa-11eb-984f-e7c6e0038e87
health: HEALTH_WARN
1 MDSs are read only
2 MDSs behind on trimming
services:
mon: 3 daemons, quorum rke-sh1-2,rke-sh1-1,rke-sh1-3 (age 17h)
mgr: rke-sh1-1.qskoyj(active, since 17h), standbys: rke-sh1-2.lxmguj, rke-sh1-3.ckunvo
mds: 1/1 daemons up, 1 standby, 1 hot standby
osd: 18 osds: 18 up (since 17h), 18 in (since 20h)
rgw: 6 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 849 pgs
objects: 10.10M objects, 5.3 TiB
usage: 11 TiB used, 15 TiB / 26 TiB avail
pgs: 849 active+clean
io:
client: 35 KiB/s rd, 1.0 MiB/s wr, 302 op/s rd, 165 op/s wr
# ceph mds stat
cephfs:1 {0=cephfs.rke-sh1-2.isqjza=up:active} 1 up:standby-replay 1 up:standby
Have you got an idea on what could be my next steps to bring the cluster healthy ?
Help will very be appreciated.
Thank a lot for your feedback.
Best Regards,
Edouard FAZENDA
Technical Support
Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40
<https://www.csti.ch/> www.csti.ch
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Eugen, We have followed the workaround here : https://tracker.ceph.com/issues/58082#note-11 And the cluster goes healthy, K8S workload are back. # ceph status cluster: id: fcb373ce-7aaa-11eb-984f-e7c6e0038e87 health: HEALTH_OK services: mon: 3 daemons, quorum rke-sh1-2,rke-sh1-1,rke-sh1-3 (age 87m) mgr: rke-sh1-2.lxmguj(active, since 2h), standbys: rke-sh1-3.ckunvo, rke-sh1-1.qskoyj mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 18 osds: 18 up (since 88m), 18 in (since 24h) rgw: 6 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 737 pgs objects: 9.98M objects, 4.8 TiB usage: 10 TiB used, 16 TiB / 26 TiB avail pgs: 737 active+clean io: client: 226 MiB/s rd, 208 MiB/s wr, 109 op/s rd, 272 op/s wr progress: Global Recovery Event (17m) [========================....] (remaining: 2m) # ceph fs status cephfs - 33 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.rke-sh1-1.ojmpnk Reqs: 51 /s 103k 97.7k 11.2k 20.7k 0-s standby-replay cephfs.rke-sh1-2.isqjza Evts: 30 /s 0 0 0 0 POOL TYPE USED AVAIL cephfs_metadata metadata 49.9G 6685G cephfs_data data 8423G 6685G STANDBY MDS cephfs.rke-sh1-3.vdicdn MDS version: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable) # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 ssd 1.45549 1.00000 1.5 TiB 562 GiB 557 GiB 13 MiB 4.6 GiB 929 GiB 37.68 0.98 86 up 4 ssd 1.45549 1.00000 1.5 TiB 566 GiB 561 GiB 9.1 MiB 4.9 GiB 925 GiB 37.95 0.98 94 up 7 ssd 1.45549 1.00000 1.5 TiB 590 GiB 584 GiB 16 MiB 5.5 GiB 901 GiB 39.57 1.03 93 up 11 ssd 1.45549 1.00000 1.5 TiB 563 GiB 558 GiB 15 MiB 4.3 GiB 928 GiB 37.75 0.98 93 up 14 ssd 1.45549 1.00000 1.5 TiB 575 GiB 570 GiB 11 MiB 4.8 GiB 916 GiB 38.56 1.00 97 up 17 ssd 1.45549 1.00000 1.5 TiB 651 GiB 646 GiB 30 MiB 4.6 GiB 840 GiB 43.67 1.13 95 up 0 ssd 1.45549 1.00000 1.5 TiB 614 GiB 608 GiB 15 MiB 5.3 GiB 877 GiB 41.18 1.07 98 up 3 ssd 1.45549 1.00000 1.5 TiB 673 GiB 668 GiB 20 MiB 4.7 GiB 817 GiB 45.16 1.17 105 up 6 ssd 1.45549 1.00000 1.5 TiB 527 GiB 523 GiB 11 MiB 4.9 GiB 963 GiB 35.39 0.92 86 up 9 ssd 1.45549 1.00000 1.5 TiB 549 GiB 545 GiB 16 MiB 4.2 GiB 942 GiB 36.83 0.95 88 up 12 ssd 1.45549 1.00000 1.5 TiB 551 GiB 546 GiB 11 MiB 4.4 GiB 940 GiB 36.95 0.96 96 up 15 ssd 1.45549 1.00000 1.5 TiB 594 GiB 589 GiB 16 MiB 4.4 GiB 897 GiB 39.83 1.03 84 up 1 ssd 1.45549 1.00000 1.5 TiB 520 GiB 516 GiB 10 MiB 3.6 GiB 970 GiB 34.89 0.90 87 up 5 ssd 1.45549 1.00000 1.5 TiB 427 GiB 423 GiB 7.9 MiB 4.0 GiB 1.0 TiB 28.64 0.74 74 up 8 ssd 1.45549 1.00000 1.5 TiB 625 GiB 620 GiB 27 MiB 4.7 GiB 866 GiB 41.92 1.09 97 up 10 ssd 1.45549 1.00000 1.5 TiB 562 GiB 557 GiB 12 MiB 5.1 GiB 929 GiB 37.69 0.98 92 up 13 ssd 1.45549 1.00000 1.5 TiB 673 GiB 668 GiB 7.2 MiB 5.0 GiB 817 GiB 45.15 1.17 101 up 16 ssd 1.45549 1.00000 1.5 TiB 534 GiB 530 GiB 5.7 MiB 3.5 GiB 957 GiB 35.81 0.93 85 up TOTAL 26 TiB 10 TiB 10 TiB 254 MiB 82 GiB 16 TiB 38.59 MIN/MAX VAR: 0.74/1.17 STDDEV: 3.88 Thanks for the help ! Best Regards, Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 www.csti.ch -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: vendredi, 23 février 2024 15:05 To: Edouard FAZENDA <e.fazenda@csti.ch> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS in ReadOnly and 2 MDS behind on trimming
2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x1 object, errno -22 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 mds.0.12487 unhandled write error (22) Invalid argument, force readonly...
Was your cephfs metadata pool full? This tracker (https://tracker.ceph.com/issues/52260) sounds very similar but I don't see a solution for it. Zitat von Edouard FAZENDA <e.fazenda@csti.ch>:
Hi Eugen,
Thanks for the reply, really appreciate
The first command , just hang with no output # cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect
The second command
# cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect Overall journal integrity: OK
root@rke-sh1-2:~# cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mds.cephfs.rke-sh1-2.isqjza -- Logs begin at Fri 2024-02-23 04:49:32 UTC, end at Fri 2024-02-23 13:08:22 UTC. -- Feb 23 07:46:46 rke-sh1-2 bash[1058012]: ignoring --setuser ceph since I am not root Feb 23 07:46:46 rke-sh1-2 bash[1058012]: ignoring --setgroup ceph since I am not root Feb 23 07:46:46 rke-sh1-2 bash[1058012]: starting mds.cephfs.rke-sh1-2.isqjza at Feb 23 08:15:06 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:06.371+0000 7fbc17dd9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. Feb 23 08:15:13 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x1 object, errno -22 Feb 23 08:15:13 rke-sh1-2 bash[1058012]: debug 2024-02-23T08:15:13.155+0000 7fbc145d2700 -1 mds.0.12487 unhandled write error (22) Invalid argument, force readonly... Feb 23 10:20:36 rke-sh1-2 bash[1058012]: debug 2024-02-23T10:20:36.309+0000 7fbc17dd9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request.
root@rke-sh1-3:~# cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mds.cephfs.rke-sh1-3.vdicdn -- Logs begin at Fri 2024-02-23 06:59:48 UTC, end at Fri 2024-02-23 13:09:18 UTC. -- Feb 23 07:46:46 rke-sh1-3 bash[2901]: ignoring --setuser ceph since I am not root Feb 23 07:46:46 rke-sh1-3 bash[2901]: ignoring --setgroup ceph since I am not root Feb 23 07:46:46 rke-sh1-3 bash[2901]: starting mds.cephfs.rke-sh1-3.vdicdn at Feb 23 10:25:51 rke-sh1-3 bash[2901]: ignoring --setuser ceph since I am not root Feb 23 10:25:51 rke-sh1-3 bash[2901]: ignoring --setgroup ceph since I am not root Feb 23 10:25:51 rke-sh1-3 bash[2901]: starting mds.cephfs.rke-sh1-3.vdicdn at
debug2: channel 0: request window-change confirm 0 debug3: send packet: type 98 -- Logs begin at Fri 2024-02-23 00:24:42 UTC, end at Fri 2024-02-23 13:09:55 UTC. -- Feb 23 09:29:10 rke-sh1-1 bash[786820]: tcmalloc: large alloc 1073750016 bytes == 0x5598512de000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41da6> Feb 23 09:29:19 rke-sh1-1 bash[786820]: tcmalloc: large alloc 2147491840 bytes == 0x559891ae0000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41db3> Feb 23 09:29:26 rke-sh1-1 bash[786820]: tcmalloc: large alloc 2147491840 bytes == 0x559951ae4000 @ 0x7fb426636760 0x7fb426657c64 0x5597c1ccaaba 0x7fb41bc04218 0x7fb41bc0ed5b 0x7fb41bbfeda4 0x7fb41da6> Feb 23 09:29:27 rke-sh1-1 bash[786820]: debug 2024-02-23T09:29:27.928+0000 7fb416d63700 -1 asok(0x5597c3904000) AdminSocket: error writing response length (32) Broken pipe Feb 23 12:35:53 rke-sh1-1 bash[786820]: ignoring --setuser ceph since I am not root Feb 23 12:35:53 rke-sh1-1 bash[786820]: ignoring --setgroup ceph since I am not root Feb 23 12:35:53 rke-sh1-1 bash[786820]: starting mds.cephfs.rke-sh1-1.ojmpnk at
The logs of the MDS are in verbose 20 , do you want me to provide on a archive ?
Is there a way to compact all the logs ?
Best Regards,
Edouard FAZENDA Technical Support
Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40
www.csti.ch
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: vendredi, 23 février 2024 12:50 To: ceph-users@ceph.io Subject: [ceph-users] Re: MDS in ReadOnly and 2 MDS behind on trimming
Hi,
the mds log should contain information why it goes into read-only mode. Just a few weeks ago I helped a user with a broken CephFS (MDS went into read-only mode because of missing objects in the journal). Can you check the journal status:
# cephfs-journal-tool --rank=cephfs:0 --journal=mdlog journal inspect
# cephfs-journal-tool --rank=cephfs:0 --journal=purge_queue journal inspect
and also share the logs.
Thanks, Eugen
Zitat von Edouard FAZENDA <e.fazenda@csti.ch>:
Dear Ceph Community,
I am having an issue with my Ceph Cluster , there were several osd crashing but now active and recovery finished and now the CephFS filesystem cannot be access by clients in RW (K8S worklod) as the 1 MDS is in Read-Only and 2 are being on trimming
The cephfs seems to have volume OK
The trimming process seems not going further, maybe stuck ?
We are running 3 hosts using ceph Pacific version 16.2.1
Here some logs on the situation :
ceph versions
{
"mon": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"mgr": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"osd": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 18
},
"mds": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3
},
"rgw": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 6
},
"overall": {
"ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 33
}
}
ceph orch ps
NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID
crash.rke-sh1-1 rke-sh1-1 running (21h) 36s ago 21h - 16.2.1 c757e4a3636b e8652edb2b49
crash.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 20M - 16.2.1 c757e4a3636b a1249a605ee0
crash.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b 026667bc1776
mds.cephfs.rke-sh1-1.ojmpnk rke-sh1-1 running (18h) 36s ago 4M - 16.2.1 c757e4a3636b 9b4c2b08b759
mds.cephfs.rke-sh1-2.isqjza rke-sh1-2 running (18h) 3m ago 23M - 16.2.1 c757e4a3636b 71681a5f34d3
mds.cephfs.rke-sh1-3.vdicdn rke-sh1-3 running (17h) 36s ago 3M - 16.2.1 c757e4a3636b e89946ad6b7e
mgr.rke-sh1-1.qskoyj rke-sh1-1 running (21h) 36s ago 2y *:8082 *:9283 16.2.1 c757e4a3636b 7ce7cfbb3e55
mgr.rke-sh1-2.lxmguj rke-sh1-2 running (21h) 3m ago 22M *:8082 *:9283 16.2.1 c757e4a3636b 5a0025adfd46
mgr.rke-sh1-3.ckunvo rke-sh1-3 running (17h) 36s ago 6M *:8082 *:9283 16.2.1 c757e4a3636b 2fcaf18f3218
mon.rke-sh1-1 rke-sh1-1 running (20h) 36s ago 20h - 16.2.1 c757e4a3636b c0a90103cabc
mon.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 3M - 16.2.1 c757e4a3636b f4b32ba4466b
mon.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b d5e44c245998
osd.0 rke-sh1-2 running (20h) 3m ago 2y - 16.2.1 c757e4a3636b 7b0e69942c15
osd.1 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 4451654d9a2d
osd.10 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 3f9d5f95e284
osd.11 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b db1cc6d2e37f
osd.12 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b de416c1ef766
osd.13 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 25a281cc5a9b
osd.14 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 62f25ba61667
osd.15 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b d3514d823c45
osd.16 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b bba857759bfe
osd.17 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 59281d4bb3d0
osd.2 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 418041b5e60d
osd.3 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 04a0e29d5623
osd.4 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b 1cc78a5153d3
osd.5 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 39a4b11e31fb
osd.6 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 2f218ffb566e
osd.7 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b cf761fbe4d5f
osd.8 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b f9f85480e800
osd.9 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 664c54ff46d2
rgw.default.rke-sh1-1.dgucwl rke-sh1-1 running (21h) 36s ago 22M *:8000 16.2.1 c757e4a3636b f03212b955a7
rgw.default.rke-sh1-1.vylchc rke-sh1-1 running (21h) 36s ago 22M *:8001 16.2.1 c757e4a3636b da486ce43fe5
rgw.default.rke-sh1-2.dfhhfw rke-sh1-2 running (21h) 3m ago 2y *:8000 16.2.1 c757e4a3636b ef4089d0aef2
rgw.default.rke-sh1-2.efkbum rke-sh1-2 running (21h) 3m ago 2y *:8001 16.2.1 c757e4a3636b 9e053d5a2f7b
rgw.default.rke-sh1-3.krfgey rke-sh1-3 running (17h) 36s ago 9M *:8001 16.2.1 c757e4a3636b 45cd3d75edd3
rgw.default.rke-sh1-3.pwdbmp rke-sh1-3 running (17h) 36s ago 9M *:8000 16.2.1 c757e4a3636b e2710265a7f4
ceph health detail
HEALTH_WARN 1 MDSs are read only; 2 MDSs behind on trimming
[WRN] MDS_READ_ONLY: 1 MDSs are read only
mds.cephfs.rke-sh1-2.isqjza(mds.0): MDS in read-only mode
[WRN] MDS_TRIM: 2 MDSs behind on trimming
mds.cephfs.rke-sh1-2.isqjza(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149
mds.cephfs.rke-sh1-1.ojmpnk(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149
root@rke-sh1-1:~# ceph fs status
cephfs - 27 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.rke-sh1-2.isqjza Reqs: 8 /s 85.2k 53.2k 1742 101
0-s standby-replay cephfs.rke-sh1-1.ojmpnk Evts: 0 /s 52.2k 20.2k 1737 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 1109G 6082G
cephfs_data data 8419G 6082G
STANDBY MDS
cephfs.rke-sh1-3.vdicdn
MDS version: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
ceph status
cluster:
id: fcb373ce-7aaa-11eb-984f-e7c6e0038e87
health: HEALTH_WARN
1 MDSs are read only
2 MDSs behind on trimming
services:
mon: 3 daemons, quorum rke-sh1-2,rke-sh1-1,rke-sh1-3 (age 17h)
mgr: rke-sh1-1.qskoyj(active, since 17h), standbys: rke-sh1-2.lxmguj, rke-sh1-3.ckunvo
mds: 1/1 daemons up, 1 standby, 1 hot standby
osd: 18 osds: 18 up (since 17h), 18 in (since 20h)
rgw: 6 daemons active (3 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 11 pools, 849 pgs
objects: 10.10M objects, 5.3 TiB
usage: 11 TiB used, 15 TiB / 26 TiB avail
pgs: 849 active+clean
io:
client: 35 KiB/s rd, 1.0 MiB/s wr, 302 op/s rd, 165 op/s wr
# ceph mds stat
cephfs:1 {0=cephfs.rke-sh1-2.isqjza=up:active} 1 up:standby-replay 1 up:standby
Have you got an idea on what could be my next steps to bring the cluster healthy ?
Help will very be appreciated.
Thank a lot for your feedback.
Best Regards,
Edouard FAZENDA
Technical Support
Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40
<https://www.csti.ch/> www.csti.ch
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Edouard FAZENDA
-
Eugen Block