Also a question, how can I identify this issue?
I have 3 high performing cluster with nvme for wal and rocksdb backed with ssds
(3ssd/1nvme) and many months ago smashing the nvmes. We are using at as objectstore. I
would assume the issue is the same as with our old cluster, but is there a way to I can be
sure about this?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com
---------------------------------------------------
-----Original Message-----
From: Szabo, Istvan (Agoda)
Sent: Tuesday, May 18, 2021 1:28 PM
To: Igor Fedotov <ifedotov(a)suse.de>
Cc: ceph-users(a)ceph.io
Subject: RE: [ceph-users] Re: Pool has been deleted before snaptrim finished
Thank you Igor your help, I've done on the smashed SSDs seems like finally the cluster
come back to normal.
How can I avoid this situation?
Should I use buffered_io or not?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com
---------------------------------------------------
-----Original Message-----
From: Igor Fedotov <ifedotov(a)suse.de>
Sent: Monday, May 17, 2021 10:36 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>om>; ceph-users(a)ceph.io
Subject: [ceph-users] Re: Pool has been deleted before snaptrim finished
Highly likely you're facing "degraded" RocksDB caused by bulk data removal.
This applies to both your original snaptrimm issue and the current flapping OSDs.
Following tickets reffer to snaptrim issue:
https://tracker.ceph.com/issues/50511 https://tracker.ceph.com/issues/47446
As a workaround you might want to compact DBs for every OSD using ceph-kvstore-tool
(offline) or ceph admin daemon's compact command (online).
Offline compaction is preferred as online one might be no-op undere some circumstanses.
Thanks,
Igor
On 5/17/2021 4:31 PM, Szabo, Istvan (Agoda) wrote:
Hi,
We decided to delete the pool before the snaptrim finished after 4 days waiting.
Now we have bigger issue, many osd started to flap, 2 of them cannot even restart due
after.
Did some bluestore fsck on the not started osds and has many messages like this inside:
2021-05-17 18:37:07.176203 7f416d20bec0 10 stupidalloc
0x0x564e4e804f50 init_add_free 0x482d0778000~4000
2021-05-17 18:37:07.176204 7f416d20bec0 10 freelist enumerate_next
0x482d0784000~4000
2021-05-17 18:37:07.176204 7f416d20bec0 10 stupidalloc
0x0x564e4e804f50 init_add_free 0x482d0784000~4000
2021-05-17 18:37:07.176205 7f416d20bec0 10 freelist enumerate_next
0x482d078c000~c000
2021-05-17 18:37:07.176206 7f416d20bec0 10 stupidalloc
0x0x564e4e804f50 init_add_free 0x482d078c000~c000
[root@hk-cephosd-2002 ~]# tail -f /tmp/ceph-osd-44-fsck.log
2021-05-17 18:39:16.466967 7f416d20bec0 20 bluefs _read_random read
buffered 0x2cd6e8f~ed6 of 1:0x372e0700000+4200000
2021-05-17 18:39:16.467154 7f416d20bec0 20 bluefs _read_random got
3798
2021-05-17 18:39:16.467179 7f416d20bec0 10 bluefs _read_random h
0x564e4e658500 0x24d6e35~ee2 from file(ino 216551 size 0x43a382d mtime
2021-05-17 13:21:19.839668 bdev 1 allocated 4400000 extents
[1:0x35bc7c00000+4400000])
2021-05-17 18:39:16.467186 7f416d20bec0 20 bluefs _read_random read
buffered 0x24d6e35~ee2 of 1:0x35bc7c00000+4400000
2021-05-17 18:39:16.467409 7f416d20bec0 20 bluefs _read_random got
3810
and
uh oh, missing shared_blob
I've set back buffered_io to false back because when restart the osds always had to
wait to fix degraded pgs.
Many of the SSDs are smashing at the moment on 100% and don't really
know what to do to stop the process and bring back the 2 ssds :/
Some paste:
https://justpaste.it/9bj3a
Some metric (each column is 1 server metric, total 3 servers):
How it is smashing the ssds:
https://i.ibb.co/x3xm0Rj/ssds.png IOWAIT
Super high due to ssd utilization:
https://i.ibb.co/683TR9y/iowait.png
Capacity seems coming back:
https://i.ibb.co/mz4Lq2r/space.png
Thank you the help.
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io