31 Dec
2020
31 Dec
'20
4:16 a.m.
Hello Ceph Users,
Since upgrading from Nautilus to Octopus ( cluster started in luminous ) I have been
trying to debug why the RocksDB/WAL is maxing out the SSD drives. ( QD > 32, 12000 read
IOPS, 200 write IOPS ).
The omap upgrade on migration was disabled initially but I reenabled it and restarted all
OSD's. This completed without issue.
I have increased the memory target from 4 to 6GB per OSD but it doesn't look like it
is using it all anyway ( based on top ).
I have offline compacted all OSDs. This seems to help for about 4-6 hours ( backfilling is
occuring - maybe this triggers it? ).
RGW garbage collection is upto date.
Pg_log on some PG's are high due to them not being in a clean state ( 8% PGs > 3000
) remainder of PG's I have reduced to 500 logs - no change.
I've been working on this issue for days not without much luck. Nothing in the logs
indicates a major issue.
The client impact is a major reduction in speed.
{
"mon": {
"ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus
(stable)": 5
},
"mgr": {
"ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus
(stable)": 1
},
"osd": {
"ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus
(stable)": 18,
"ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus
(stable)": 280
},
"mds": {},
"rgw": {
"ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus
(stable)": 2
},
"tcmu-runner": {
"ceph version 14.2.13-450-g65ea1b614d
(65ea1b614db8b6d10f334a8ff67c4de97f73bcbf) nautilus (stable)": 2
},
"overall": {
"ceph version 14.2.13-450-g65ea1b614d
(65ea1b614db8b6d10f334a8ff67c4de97f73bcbf) nautilus (stable)": 2,
"ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus
(stable)": 18,
"ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus
(stable)": 288
}
}
Any assistance in debugging would be greatly helpful.
Glen
This e-mail is intended solely for the benefit of the addressee(s) and any other named
recipient. It is confidential and may contain legally privileged or confidential
information. If you are not the recipient, any use, distribution, disclosure or copying of
this e-mail is prohibited. The confidentiality and legal privilege attached to this
communication is not waived or lost by reason of the mistaken transmission or delivery to
you. If you have received this e-mail in error, please notify us immediately.