22 Feb
2024
22 Feb
'24
11:36 a.m.
We have 6 node ( 3 OSD-node and 3 service node), t2/3 OSD nodes was powered off and we got
big problem
pls check ceph-s result below
now we cannot start mds service, ( we tried to start but it stopped after 2 minute)
Now my application cannot access to NFS exported Folder
What should we do
[root@cephgw01 /]# ceph -s
cluster:
id: 258af72a-cff3-11eb-a261-d4f5ef25154c
health: HEALTH_WARN
3 failed cephadm daemon(s)
1 filesystem is degraded
insufficient standby MDS daemons available
1 nearfull osd(s)
Low space hindering backfill (add storage if this doesn't resolve itself):
21 pgs backfill_toofull
15 pool(s) nearfull
11 daemons have recently crashed
services:
mon: 6 daemons, quorum
cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h)
mgr: cephgw01.vwoffq(active, since 17h), standbys: cephgw02.nauphz,
cephgw03.aipvii
mds: 1/1 daemons up
osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402 remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones)
tcmu-runner: 18 daemons active (2 hosts)
data:
volumes: 0/1 healthy, 1 recovering
pools: 15 pools, 1457 pgs
objects: 36.87M objects, 25 TiB
usage: 75 TiB used, 41 TiB / 116 TiB avail
pgs: 17759672/110607480 objects misplaced (16.056%)
1055 active+clean
363 active+remapped+backfill_wait
18 active+remapped+backfilling
14 active+remapped+backfill_toofull
7 active+remapped+backfill_wait+backfill_toofull
io:
client: 2.0 MiB/s rd, 395 KiB/s wr, 73 op/s rd, 19 op/s wr
recovery: 32 MiB/s, 45 objects/s
22 Feb
22 Feb
12:31 p.m.
What does the MDS log when it crashes?
Zitat von nguyenvandiep(a)baoviet.com.vn:
We have 6 node ( 3 OSD-node and 3 service node), t2/3
OSD nodes was
powered off and we got big problem
pls check ceph-s result below
now we cannot start mds service, ( we tried to start but it stopped
after 2 minute)
Now my application cannot access to NFS exported Folder
What should we do
[root@cephgw01 /]# ceph -s
cluster:
id: 258af72a-cff3-11eb-a261-d4f5ef25154c
health: HEALTH_WARN
3 failed cephadm daemon(s)
1 filesystem is degraded
insufficient standby MDS daemons available
1 nearfull osd(s)
Low space hindering backfill (add storage if this
doesn't resolve itself): 21 pgs backfill_toofull
15 pool(s) nearfull
11 daemons have recently crashed
services:
mon: 6 daemons, quorum
cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h)
mgr: cephgw01.vwoffq(active, since 17h), standbys:
cephgw02.nauphz, cephgw03.aipvii
mds: 1/1 daemons up
osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402
remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones)
tcmu-runner: 18 daemons active (2 hosts)
data:
volumes: 0/1 healthy, 1 recovering
pools: 15 pools, 1457 pgs
objects: 36.87M objects, 25 TiB
usage: 75 TiB used, 41 TiB / 116 TiB avail
pgs: 17759672/110607480 objects misplaced (16.056%)
1055 active+clean
363 active+remapped+backfill_wait
18 active+remapped+backfilling
14 active+remapped+backfill_toofull
7 active+remapped+backfill_wait+backfill_toofull
io:
client: 2.0 MiB/s rd, 395 KiB/s wr, 73 op/s rd, 19 op/s wr
recovery: 32 MiB/s, 45 objects/s
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:52 p.m.
There a couple of ways, find your MDS daemon with:
ceph fs status -> should show you the to-be-active MDS
On that host run:
cephadm logs --name mds.{MDS}
or alternatively:
cephadm ls --no-detail | grep mds
journalctl -u ceph-{FSID}(a)mds.{MDS} --no-pager > {MDS}.log
Zitat von nguyenvandiep(a)baoviet.com.vn:
How can we get log of MDS, pls guide me T_T
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:08 p.m.
it suck too long log, could you pls guide me how to grep/filter important things in logs ?
1:15 p.m.
If it crashes after two minutes you have your time window to look for.
Restart the mds daemon and capture everything after that until the
crash.
Zitat von nguyenvandiep(a)baoviet.com.vn:
it suck too long log, could you pls guide me how to
grep/filter
important things in logs ?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Feb
23 Feb
5:11 a.m.
Could you pls help me explain the status of volume: recovering ? what is it ? and do we
need to wait for volume recovery progress finished ??
10:59 a.m.
You still haven't provided any details (logs) of what happened. The
short excerpt from yesterday isn't useful as it only shows the startup
of the daemon.
Zitat von nguyenvandiep(a)baoviet.com.vn:
Could you pls help me explain the status of volume:
recovering ?
what is it ? and do we need to wait for volume recovery progress
finished ??
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:55 a.m.
https://drive.google.com/file/d/1OIN5O2Vj0iWfEMJ2fyHN_xV6fpknBmym/view?usp=…
Pls check my mds log which generate by command
cephadm logs --name mds.cephfs.cephgw02.qqsavr --fsid 258af72a-cff3-11eb-a261-d4f5ef25154c
12:18 p.m.
This seems to be the relevant stack trace:
---snip---
Feb 23 15:18:39 cephgw02 conmon[2158052]: debug -1>
2024-02-23T08:18:39.609+0000 7fccc03c0700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h:
In function 'std::ostream& operator<<(std::ostream&, const
ClientMetricType&)' thread 7fccc03c0700 time
2024-02-23T08:18:39.609581+0000
Feb 23 15:18:39 cephgw02 conmon[2158052]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h:
56: ceph_abort_msg("abort()
called")
Feb 23 15:18:39 cephgw02 conmon[2158052]:
Feb 23 15:18:39 cephgw02 conmon[2158052]: ceph version 16.2.4
(3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
Feb 23 15:18:39 cephgw02 conmon[2158052]: 1: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0xe5)
[0x7fccc9021cdc]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 2:
(operator<<(std::ostream&, ClientMetricType const&)+0x10e)
[0x7fccc92a642e]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 3:
(MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fccc92a6601]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 4:
(DispatchQueue::pre_dispatch(boost::intrusive_ptr<Message>
const&)+0x710) [0x7fccc9259c30]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 5:
(DispatchQueue::entry()+0xdeb) [0x7fccc925b69b]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 6:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7fccc930bb71]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 7:
/lib64/libpthread.so.0(+0x814a) [0x7fccc7dc314a]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 8: clone()
Feb 23 15:18:39 cephgw02 conmon[2158052]:
Feb 23 15:18:39 cephgw02 conmon[2158052]: debug 0>
2024-02-23T08:18:39.610+0000 7fccc03c0700 -1 *** Caught signal
(Aborted) **
Feb 23 15:18:39 cephgw02 conmon[2158052]: in thread 7fccc03c0700
thread_name:ms_dispatch
Feb 23 15:18:39 cephgw02 conmon[2158052]:
Feb 23 15:18:39 cephgw02 conmon[2158052]: ceph version 16.2.4
(3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
Feb 23 15:18:39 cephgw02 conmon[2158052]: 1:
/lib64/libpthread.so.0(+0x12b20) [0x7fccc7dcdb20]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 2: gsignal()
Feb 23 15:18:39 cephgw02 conmon[2158052]: 3: abort()
Feb 23 15:18:39 cephgw02 conmon[2158052]: 4: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0x1b6)
[0x7fccc9021dad]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 5: (opera
Feb 23 15:18:39 cephgw02 conmon[2158052]: tor<<(std::ostream&,
ClientMetricType const&)+0x10e) [0x7fccc92a642e]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 6:
(MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fccc92a6601]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 7:
(DispatchQueue::pre_dispatch(boost::intrusive_ptr<Message>
const&)+0x710) [0x7fccc9259c30]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 8:
(DispatchQueue::entry()+0xdeb) [0x7fccc925b69b]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 9:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7fccc930bb71]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 10:
/lib64/libpthread.so.0(+0x814a) [0x7fccc7dc314a]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 11: clone()
---snip---
But I can't really help here, hopefully someone else can chime in and
interpret it.
Zitat von nguyenvandiep(a)baoviet.com.vn:
https://drive.google.com/file/d/1OIN5O2Vj0iWfEMJ2fyHN_xV6fpknBmym/view?usp=…
Pls check my mds log which generate by command
cephadm logs --name mds.cephfs.cephgw02.qqsavr --fsid
258af72a-cff3-11eb-a261-d4f5ef25154c
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5 p.m.
Hi,
The problem seems to come from the clients (reconnect).
Test by disabling metrics on all clients:
echo Y > /sys/module/ceph/parameters/disable_send_metrics
________________________________________________________
Cordialement,
*David CASIER*
________________________________________________________
Le ven. 23 févr. 2024 à 10:20, Eugen Block <eblock(a)nde.ag> a écrit :
This seems to be the relevant stack trace:
---snip---
Feb 23 15:18:39 cephgw02 conmon[2158052]: debug -1>
2024-02-23T08:18:39.609+0000 7fccc03c0700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h:
In function 'std::ostream& operator<<(std::ostream&, const
ClientMetricType&)' thread 7fccc03c0700 time
2024-02-23T08:18:39.609581+0000
Feb 23 15:18:39 cephgw02 conmon[2158052]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h:
56: ceph_abort_msg("abort()
called")
Feb 23 15:18:39 cephgw02 conmon[2158052]:
Feb 23 15:18:39 cephgw02 conmon[2158052]: ceph version 16.2.4
(3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
Feb 23 15:18:39 cephgw02 conmon[2158052]: 1: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0xe5)
[0x7fccc9021cdc]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 2:
(operator<<(std::ostream&, ClientMetricType const&)+0x10e)
[0x7fccc92a642e]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 3:
(MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fccc92a6601]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 4:
(DispatchQueue::pre_dispatch(boost::intrusive_ptr<Message>
const&)+0x710) [0x7fccc9259c30]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 5:
(DispatchQueue::entry()+0xdeb) [0x7fccc925b69b]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 6:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7fccc930bb71]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 7:
/lib64/libpthread.so.0(+0x814a) [0x7fccc7dc314a]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 8: clone()
Feb 23 15:18:39 cephgw02 conmon[2158052]:
Feb 23 15:18:39 cephgw02 conmon[2158052]: debug 0>
2024-02-23T08:18:39.610+0000 7fccc03c0700 -1 *** Caught signal
(Aborted) **
Feb 23 15:18:39 cephgw02 conmon[2158052]: in thread 7fccc03c0700
thread_name:ms_dispatch
Feb 23 15:18:39 cephgw02 conmon[2158052]:
Feb 23 15:18:39 cephgw02 conmon[2158052]: ceph version 16.2.4
(3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
Feb 23 15:18:39 cephgw02 conmon[2158052]: 1:
/lib64/libpthread.so.0(+0x12b20) [0x7fccc7dcdb20]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 2: gsignal()
Feb 23 15:18:39 cephgw02 conmon[2158052]: 3: abort()
Feb 23 15:18:39 cephgw02 conmon[2158052]: 4: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0x1b6)
[0x7fccc9021dad]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 5: (opera
Feb 23 15:18:39 cephgw02 conmon[2158052]: tor<<(std::ostream&,
ClientMetricType const&)+0x10e) [0x7fccc92a642e]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 6:
(MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fccc92a6601]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 7:
(DispatchQueue::pre_dispatch(boost::intrusive_ptr<Message>
const&)+0x710) [0x7fccc9259c30]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 8:
(DispatchQueue::entry()+0xdeb) [0x7fccc925b69b]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 9:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7fccc930bb71]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 10:
/lib64/libpthread.so.0(+0x814a) [0x7fccc7dc314a]
Feb 23 15:18:39 cephgw02 conmon[2158052]: 11: clone()
---snip---
But I can't really help here, hopefully someone else can chime in and
interpret it.
Zitat von nguyenvandiep(a)baoviet.com.vn:
https://drive.google.com/file/d/1OIN5O2Vj0iWfEMJ2fyHN_xV6fpknBmym/view?usp=…
> Pls check my mds log which generate
by command
> cephadm logs --name
mds.cephfs.cephgw02.qqsavr --fsid
> 258af72a-cff3-11eb-a261-d4f5ef25154c
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:33 p.m.
Hi David,
Could you pls helo me understand,
Does it affect to RGW service ? And if something go bad, how can i rollback ?
6:37 p.m.
And we dont have parameter folder
cd /sys/module/ceph/
[root@cephgw01 ceph]# ls
coresize holders initsize initstate notes refcnt rhelversion sections srcversion
taint uevent
My Ceph is 16.2.4
6:56 p.m.
look at ALL cephfs kernel clients (no effect on RGW)
Le ven. 23 févr. 2024 à 16:38, <nguyenvandiep(a)baoviet.com.vn> a écrit :
And we dont have parameter folder
cd /sys/module/ceph/
[root@cephgw01 ceph]# ls
coresize holders initsize initstate notes refcnt rhelversion
sections srcversion taint uevent
My Ceph is 16.2.4
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:30 a.m.
Can you send sudo ceph -s and sudo ceph health detail
Sent from Bloomberg Professional for iPhone
----- Original Message -----
From: nguyenvandiep(a)baoviet.com.vn
To: ceph-users(a)ceph.io
At: 02/23/24 20:27:53 UTC-05:00
Could you pls guide me more detail :( im very newbie in Ceph :(
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:59 a.m.
Hi Mathew
Pls chekc my ceph -s
ceph -s
cluster:
id: 258af72a-cff3-11eb-a261-d4f5ef25154c
health: HEALTH_WARN
3 failed cephadm daemon(s)
1 filesystem is degraded
insufficient standby MDS daemons available
1 nearfull osd(s)
Low space hindering backfill (add storage if this doesn't resolve
itself):
21 pgs backfill_toofull
15 pool(s) nearfull
11 daemons have recently crashed
services:
mon: 6 daemons, quorum
cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h)
mgr: cephgw01.vwoffq(active, since 17h), standbys: cephgw02.nauphz,
cephgw03.aipvii
mds: 1/1 daemons up
osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402 remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones)
tcmu-runner: 18 daemons active (2 hosts)
data:
volumes: 0/1 healthy, 1 recovering
pools: 15 pools, 1457 pgs
objects: 36.87M objects, 25 TiB
usage: 75 TiB used, 41 TiB / 116 TiB avail
pgs: 17759672/110607480 objects misplaced (16.056%)
1055 active+clean
363 active+remapped+backfill_wait
18 active+remapped+backfilling
14 active+remapped+backfill_toofull
7
12:19 p.m.
Do you have the possibility to stop/unmount cephfs clients ?
If so, do that and restart the MDS.
It should restart.
Have the clients restart one by one and check that the MDS does not crash
(by monitoring the logs)
________________________________________________________
Cordialement,
*David CASIER*
*Ligne directe: +33(0) 9 72 61 98 29*
________________________________________________________
Le sam. 24 févr. 2024 à 10:01, <nguyenvandiep(a)baoviet.com.vn> a écrit :
Hi Mathew
Pls chekc my ceph -s
ceph -s
cluster:
id: 258af72a-cff3-11eb-a261-d4f5ef25154c
health: HEALTH_WARN
3 failed cephadm daemon(s)
1 filesystem is degraded
insufficient standby MDS daemons available
1 nearfull osd(s)
Low space hindering backfill (add storage if this doesn't
resolve itself):
21 pgs backfill_toofull
15 pool(s) nearfull
11 daemons have recently crashed
services:
mon: 6 daemons, quorum
cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h)
mgr: cephgw01.vwoffq(active, since 17h), standbys:
cephgw02.nauphz,
cephgw03.aipvii
mds: 1/1 daemons up
osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402
remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones)
tcmu-runner: 18 daemons active (2 hosts)
data:
volumes: 0/1 healthy, 1 recovering
pools: 15 pools, 1457 pgs
objects: 36.87M objects, 25 TiB
usage: 75 TiB used, 41 TiB / 116 TiB avail
pgs: 17759672/110607480 objects misplaced (16.056%)
1055 active+clean
363 active+remapped+backfill_wait
18 active+remapped+backfilling
14 active+remapped+backfill_toofull
7
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:54 p.m.
Hi David,
I ll follow your suggestion. Do you have Telegram ? If yes, could you pls add my Telegram,
+84989177619. Thank you so much
7:53 p.m.
if rebalancing tasks have been launched it's not a big deal, but I don't
think it's the priority.
The priority being to get the MDS back on its feet.
I haven't seen an answer to this question: can you stop/unmount cephfs
clients or not ?
There are other solutions but as you are not comfortable I am making the
simplest one on the ceph side but not the most comfortable on the business
side.
There is no one (in Vietnam?) who could help you more seriously (and in a
lasting way) ?
Le sam. 24 févr. 2024 à 15:55, <nguyenvandiep(a)baoviet.com.vn> a écrit :
Hi David,
I ll follow your suggestion. Do you have Telegram ? If yes, could you pls
add my Telegram, +84989177619. Thank you so much
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:07 p.m.
Once recovery is underway way simply restarting the RGWs should be enough to reset them
and get your object store back up.
Bloomberg doesn’t use cephfs so hopefully David’s suggestions work or if anyone else in
the community can chip in for that part.
Sent from Bloomberg Professional for iPhone
----- Original Message -----
From: David C. <david.casier(a)aevoo.fr>
To: nguyenvandiep(a)baoviet.com.vn
CC: ceph-users(a)ceph.io
At: 02/24/24 16:55:28 UTC
if rebalancing tasks have been launched it's not a big deal, but I don't
think it's the priority.
The priority being to get the MDS back on its feet.
I haven't seen an answer to this question: can you stop/unmount cephfs
clients or not ?
There are other solutions but as you are not comfortable I am making the
simplest one on the ceph side but not the most comfortable on the business
side.
There is no one (in Vietnam?) who could help you more seriously (and in a
lasting way) ?
Le sam. 24 févr. 2024 à 15:55, <nguyenvandiep(a)baoviet.com.vn> a écrit :
Hi David,
I ll follow your suggestion. Do you have Telegram ? If yes, could you pls
add my Telegram, +84989177619. Thank you so much
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:23 p.m.
It looks like you have quite a few problems I’ll try and address them one by one.
1) Looks like you had a bunch of crashes, from the ceph -s it looks like you don’t have
enough MDS daemons running for a quorum. So you’ll need to restart the crashed containers.
2) It looks like you might have an interesting crush map. Allegedly you have 41TiB of
space but you can’t finish rococering you have lots of PGs stuck as their destination is
too full. Are you running homogenous hardware or do you have different drive sizes? Are
all the weights set correctly?
One you correct item 1 you’ll need to correct item 2 to get back to a healthy spot.
Sent from Bloomberg Professional for iPhone
----- Original Message -----
From: nguyenvandiep(a)baoviet.com.vn
To: ceph-users(a)ceph.io
At: 02/24/24 09:01:22 UTC
Hi Mathew
Pls chekc my ceph -s
ceph -s
cluster:
id: 258af72a-cff3-11eb-a261-d4f5ef25154c
health: HEALTH_WARN
3 failed cephadm daemon(s)
1 filesystem is degraded
insufficient standby MDS daemons available
1 nearfull osd(s)
Low space hindering backfill (add storage if this doesn't resolve
itself):
21 pgs backfill_toofull
15 pool(s) nearfull
11 daemons have recently crashed
services:
mon: 6 daemons, quorum
cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h)
mgr: cephgw01.vwoffq(active, since 17h), standbys: cephgw02.nauphz,
cephgw03.aipvii
mds: 1/1 daemons up
osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402 remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones)
tcmu-runner: 18 daemons active (2 hosts)
data:
volumes: 0/1 healthy, 1 recovering
pools: 15 pools, 1457 pgs
objects: 36.87M objects, 25 TiB
usage: 75 TiB used, 41 TiB / 116 TiB avail
pgs: 17759672/110607480 objects misplaced (16.056%)
1055 active+clean
363 active+remapped+backfill_wait
18 active+remapped+backfilling
14 active+remapped+backfill_toofull
7
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:06 p.m.
2) It looks like you might have an interesting crush map. Allegedly you have 41TiB of
space but you can’t finish rococering you have lots of PGs stuck as their destination is
too full. Are you running homogenous hardware or do you have different drive sizes? Are
all the weights set correctly?
I suspect that the OP might not have balancing enabled, or that it isn’t functioning well.
Which a complex CRUSH topology could exacerbate.
*Temporarily* raising the full and backfillfull thresholds might help, but they would want
to be reverted after sorting out the balancing.
Please supply
`ceph osd tree`
`ceph osd df`
6:22 p.m.
There ya go.
You have 4 hosts, one of which appears to be down and have a single OSD that is so small
as to not be useful. Whatever cephgw03 is, it looks like a mistake. OSDs much smaller
than, say, 1TB often aren’t very useful.
Your pools appear to be replicated, size=3.
So each of your cephosd* hosts stores one replica of each RADOS object.
You added the 10TB spinners to only two of your hosts, which means that they’re only being
used as though they were 4TB OSDs. That’s part of what’s going on.
You want to add a 10TB spinner to cephosd02. That will help your situation
significantly.
After that, consider adding a cephosd04 host. Having at least one more failure domain
than replicas lets you better use uneven host capacities.
On Feb 24, 2024, at 10:06 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
Hi Mr Anthony,
pls check the output
https://anotepad.com/notes/s7nykdmc
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:29 p.m.
I will correct some small things:
we have 6 nodes, 3 osd node and 3 gaeway node ( which run RGW, mds and nfs service)
you r corrct, 2/3 osd node have ONE-NEW 10tib disk
About your suggestion, add another osd host, we will. But we need to end this nightmare,
my NFS folder which have 10tib data is down :(
My ratio
ceph osd dump | grep ratio
full_ratio 0.95
backfillfull_ratio 0.92
nearfull_ratio 0.85
6:30 p.m.
Add a 10tb HDD to the third node as I suggested, that will help your cluster.
On Feb 24, 2024, at 10:29 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
I will correct some small things:
we have 6 nodes, 3 osd node and 3 gaeway node ( which run RGW, mds and nfs service)
you r corrct, 2/3 osd node have ONE-NEW 10tib disk
About your suggestion, add another osd host, we will. But we need to end this nightmare,
my NFS folder which have 10tib data is down :(
My ratio
ceph osd dump | grep ratio
full_ratio 0.95
backfillfull_ratio 0.92
nearfull_ratio 0.85
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:33 p.m.
Thank you, Sir. But i think i ll wait for PG BACKFILLFULL finish, my boss is very angry
now and will not allow me to add one more disk( this action make him think that ceph would
take more time for recovering and rebalancing ). We want to wait volume recovering
progress finish
6:36 p.m.
You aren’t going to be able to finish recovery without having somewhere to recover TO.
On Feb 24, 2024, at 10:33 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
Thank you, Sir. But i think i ll wait for PG BACKFILLFULL finish, my boss is very angry
now and will not allow me to add one more disk( this action make him think that ceph would
take more time for recovering and rebalancing ). We want to wait volume recovering
progress finish
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:41 p.m.
HolySh***
First, we change the mon_max_pg_per_osd to 1000
About adding disk for cephosd02, for more detail , what is TO, sir ? I ll make
conversation with my boss. To be honest, im thinking that the volume recovery progress
will get problem...
6:44 p.m.
Your recovery is stuck because there are no OSDs that have enough space to accept data.
Your second OSD host appears to only have 9 OSDs currently, so you should be able to add a
10TB OSD there without removing anything.
That will enable data to move to all three of your 10TB OSDs.
On Feb 24, 2024, at 10:41 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
HolySh***
First, we change the mon_max_pg_per_osd to 1000
About adding disk for cephosd02, for more detail , what is TO, sir ? I ll make
conversation with my boss. To be honest, im thinking that the volume recovery progress
will get problem...
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:01 p.m.
Anthony is correct, this is what I was getting at as well when seeing your ceph -s output.
More details in the Ceph docs here if you want to understand the details of why you need
to balance your nodes.
https://docs.ceph.com/en/quincy/rados/operations/monitoring-osd-pg/
But you need to get your core rados cluster healthy, then you can pivot to trying to
bringing back your gateway services.
Sent from Bloomberg Professional for iPhone
----- Original Message -----
From: Anthony D'Atri <anthony.datri(a)gmail.com>
To: nguyenvandiep(a)baoviet.com.vn
CC: ceph-users(a)ceph.io
At: 02/24/24 15:46:24 UTC
Your recovery is stuck because there are no OSDs that have enough space to accept data.
Your second OSD host appears to only have 9 OSDs currently, so you should be able to add a
10TB OSD there without removing anything.
That will enable data to move to all three of your 10TB OSDs.
On Feb 24, 2024, at 10:41 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
HolySh***
First, we change the mon_max_pg_per_osd to 1000
About adding disk for cephosd02, for more detail , what is TO, sir ? I ll make
conversation with my boss. To be honest, im thinking that the volume recovery progress
will get problem...
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:12 p.m.
Thank you Matthew.
Im following guidance from Mr Anthony and now my recovery progress speed is much faster.
I will update my case day by day.
Thank you so much
7:17 p.m.
Sudo watch ceph -s
You should see stats on the recovery and see PGs transition from all the backfill* states
to active+clean
Once you get everything active clean then we can focus on your rgws and MDSs
Sent from Bloomberg Professional for iPhone
----- Original Message -----
From: nguyenvandiep(a)baoviet.com.vn
To: ceph-users(a)ceph.io
At: 02/24/24 16:14:12 UTC
Thank you Matthew.
Im following guidance from Mr Anthony and now my recovery progress speed is much faster.
I will update my case day by day.
Thank you so much
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:22 p.m.
Thank you so much, Matthew. Pls keep an eye on my thread.
You and Mr Anthony made my day.
7:02 p.m.
Yes, Sir.
We added 10TIb to cephosd02 node. Now the disk is IN, but DOWN state.
What should we do now :(
For additional, the recovery speed is x10 times :)
7:07 p.m.
Hi Mr Anthony,
Forget it, the osd is UP and recovery speed is x10times
Amazing
And now we just wait, right ?
7:18 p.m.
Low space hindering backfill (add storage if
this doesn't resolve itself): 21 pgs backfill_toofull
^^^ Ceph even told you what you need to do ;)
If your have recovery taking place and the numbers of misplaced objects and *full
PGs/pools keeps decreasing, then yes wait.
As for getting your MDS / CephFS going after that, I’ll defer to others.
On Feb 24, 2024, at 11:07 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
Hi Mr Anthony,
Forget it, the osd is UP and recovery speed is x10times
Amazing
And now we just wait, right ?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:35 p.m.
You also might want to increase mon_max_pg_per_osd since you have a wide spread of OSD
sizes.
Default is 250. Set it to 1000.
On Feb 24, 2024, at 10:30 AM, Anthony D'Atri
<anthony.datri(a)gmail.com> wrote:
Add a 10tb HDD to the third node as I suggested, that will help your cluster.
On Feb 24, 2024, at 10:29 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
I will correct some small things:
we have 6 nodes, 3 osd node and 3 gaeway node ( which run RGW, mds and nfs service)
you r corrct, 2/3 osd node have ONE-NEW 10tib disk
About your suggestion, add another osd host, we will. But we need to end this nightmare,
my NFS folder which have 10tib data is down :(
My ratio
ceph osd dump | grep ratio
full_ratio 0.95
backfillfull_ratio 0.92
nearfull_ratio 0.85
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
16 Mar
16 Mar
12:29 p.m.
Hello Van Diep,
I read this after you got out of trouble.
According to your ceph osd tree, it looks like your problems started when the ceph
orchestrator created osd.29 on node 'cephgw03' because it looks very unlikely that
you created a 100MB OSD on a node that's named after "GW".
You may have added the 'osds' label to the 'cephgw03' node and/or played
with the service_type:osd and/or added the cephgw03 node to the crushmap, which triggered
the creation of osd.29 by the orchestrator.
cephgw03 node being part of the 'default' root bucket, other OSDs legitimately
started to send objects to osd.29, way to small to accommodate them, PGs then becoming
'backfill_toofull'.
To get out of this situation, you could have:
$ ceph osd crush add-bucket closet root
$ ceph osd crush move cephgw03 root=closet
This would have moved 'cephgw03' node out of the 'default' root and
probably fixed your problem instantly.
Regards,
Frédéric.
-----Message original-----
De: Anthony <anthony.datri(a)gmail.com>
à: nguyenvandiep <nguyenvandiep(a)baoviet.com.vn>
Cc: ceph-users <ceph-users(a)ceph.io>
Envoyé: samedi 24 février 2024 16:24 CET
Sujet : [ceph-users] Re: [Urgent] Ceph system Down, Ceph FS volume in recovering
There ya go.
You have 4 hosts, one of which appears to be down and have a single OSD that is so small
as to not be useful. Whatever cephgw03 is, it looks like a mistake. OSDs much smaller
than, say, 1TB often aren’t very useful.
Your pools appear to be replicated, size=3.
So each of your cephosd* hosts stores one replica of each RADOS object.
You added the 10TB spinners to only two of your hosts, which means that they’re only being
used as though they were 4TB OSDs. That’s part of what’s going on.
You want to add a 10TB spinner to cephosd02. That will help your situation significantly.
After that, consider adding a cephosd04 host. Having at least one more failure domain than
replicas lets you better use uneven host capacities.
On Feb 24, 2024, at 10:06 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
Hi Mr Anthony,
pls check the output
https://anotepad.com/notes/s7nykdmc
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
24 Feb
24 Feb
6:07 p.m.
Hi Mr Anthony
pls check the output
https://anotepad.com/notes/s7nykdmc
6:12 p.m.
Hi Mr Anthony, Could you tell me more details about raising the full and backfullfull
threshold
is it
ceph tell 'osd.*' injectargs --osd-max-backfills=2 --osd-recovery-max-active=6
??
6:25 p.m.
# ceph osd dump | grep ratio
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
Read the four sections here:
https://docs.ceph.com/en/quincy/rados/operations/health-checks/#osd-out-of-…
On Feb 24, 2024, at 10:12 AM,
nguyenvandiep(a)baoviet.com.vn wrote:
Hi Mr Anthony, Could you tell me more details about raising the full and backfullfull
threshold
is it
ceph tell 'osd.*' injectargs --osd-max-backfills=2 --osd-recovery-max-active=6
??
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:03 p.m.
Hi Mathew,
1) We have 2 MDS service running before this nightmare. Now we trying to apply mds on 3
nodes, but all of them will stop within 2 minutes.
2) You are correct. We just add two 10TIB disk to cluster ( which currently have 27 x 4TIB
disk), all of them have weight 1.0
About volume recovery and pg backfill full, will volume recovery process finish after pg
backfill full process completed ?
And do you have Telegram ? If yes, could you pls add my Telegram,
+84989177619. Thank you so much
22 Feb
22 Feb
1:12 p.m.
Feb 22 13:39:43 cephgw02 conmon[1340927]: log_file
/var/lib/ceph/crash/2024-02-22T06:39:43.618845Z_78ee38bc-9115-4bc6-8c3a-4bf42284c970/log
Feb 22 13:39:43 cephgw02 conmon[1340927]: --- end dump of recent events ---
Feb 22 13:39:45 cephgw02 systemd[1]:
ceph-258af72a-cff3-11eb-a261-d4f5ef25154c(a)mds.cephfs.cephgw02.qqsavr.service: Main process
exited, code=exited, status=134/n/a
Feb 22 13:39:45 cephgw02 systemd[1]:
ceph-258af72a-cff3-11eb-a261-d4f5ef25154c(a)mds.cephfs.cephgw02.qqsavr.service: Failed with
result 'exit-code'.
Feb 22 13:39:55 cephgw02 systemd[1]:
ceph-258af72a-cff3-11eb-a261-d4f5ef25154c(a)mds.cephfs.cephgw02.qqsavr.service: Service
RestartSec=10s expired, scheduling restart.
Feb 22 13:39:55 cephgw02 systemd[1]:
ceph-258af72a-cff3-11eb-a261-d4f5ef25154c(a)mds.cephfs.cephgw02.qqsavr.service: Scheduled
restart job, restart counter is at 4.
Feb 22 13:39:55 cephgw02 systemd[1]: Stopped Ceph mds.cephfs.cephgw02.qqsavr for
258af72a-cff3-11eb-a261-d4f5ef25154c.
Feb 22 13:39:55 cephgw02 systemd[1]: Starting Ceph mds.cephfs.cephgw02.qqsavr for
258af72a-cff3-11eb-a261-d4f5ef25154c...
Feb 22 13:39:56 cephgw02 bash[1341570]:
636aedce612582726e7eb229eb63d0f641491994fc76f1de1828b61d07d6e8ac
Feb 22 13:39:56 cephgw02 systemd[1]: Started Ceph mds.cephfs.cephgw02.qqsavr for
258af72a-cff3-11eb-a261-d4f5ef25154c.
Feb 22 13:39:56 cephgw02 conmon[1341683]: debug 2024-02-22T06:39:56.151+0000 7fc4e3309780
0 set uid:gid to 167:167 (ceph:ceph)
Feb 22 13:39:56 cephgw02 conmon[1341683]: debug 2024-02-22T06:39:56.151+0000 7fc4e3309780
0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process
ceph-mds, pid 7
26 Feb
26 Feb
1:30 p.m.
Dear Mr Eugen, Mr Matthew, Mr David, Mr Anthony
My System is UP.
Thank you so much. We get many support from all of you., mazing, kindly support from Top
professional in Ceph.
Hope we have a chance to cooperate in the future. And If you travel to VietNam in future,
let me know. I ll be your local tour guide and we can get some beers.
Once again, Thank you
====the solution is=====
Try :
ceph config set mds mds_deny_all_reconnect true
and restart MDS.
After that (mds active) :
ceph config rm mds mds_deny_all_reconnect
========From Mr David====
1:34 p.m.
Glad to hear it all worked out for you!
From: nguyenvandiep(a)baoviet.com.vn At: 02/26/24 05:32:32 UTC-5:00To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: [Urgent] Ceph system Down, Ceph FS volume in recovering
Dear Mr Eugen, Mr Matthew, Mr David, Mr Anthony
My System is UP.
Thank you so much. We get many support from all of you., mazing, kindly support
from Top professional in Ceph.
Hope we have a chance to cooperate in the future. And If you travel to VietNam
in future, let me know. I ll be your local tour guide and we can get some beers.
Once again, Thank you
====the solution is=====
Try :
ceph config set mds mds_deny_all_reconnect true
and restart MDS.
After that (mds active) :
ceph config rm mds mds_deny_all_reconnect
========From Mr David====
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
64
days inactive
87
days old
49 comments
6 participants
participants (6)
-
Anthony D'Atri -
David C. -
Eugen Block -
Frédéric Nass -
Matthew Leonard (BLOOMBERG/ 120 PARK) -
nguyenvandiep＠baoviet.com.vn