The OP implies that the cluster's performance *degraded* with the Quincy upgrade.I wonder if there was a kernel change at the same time.
No, it's never been great. But it's definitely getting worse over time. That is most likely correlated with increased utilization (both in terms of space used and IOPs demanded) rather than any specific upgrades.
Ahhhhh, from the original phrasing I thought that Quincy correlated with a sharp drop.
They have the latest firmware
As per recent isdct/intelmas/sst? The web site?
I'm interested in stopping it from completely stalling out for 10-30 seconds at a time on a regular basis. The actual number of IOPs of a properly-functioning cluster is a *very* secondary concern.
Just for grins, I might suggest -- once you have a fully healthy cluster -- destroying, secure-erasing, and redeploying a few OSDs at a time within a single failure domain. How old are the OSDs? I suspect at least some might be Filestore and thus would be redeployed with BlueStore. Newer SSD controllers / models are better than older models at housekeeping over time, so the secure-erase might freshen performance.
Networking is also operating at a tiny fraction of its capacity. It looks like each server runs about 15-30 *megabits* per OSD. None of them push even 1Gbps, much less 10.
Fair enough. Layer 1 issues can still matter though.
We have tested connectivity between all twelve ceph machines with all 11 others machines with iperf3. Any two machines can iperf3 7-9Gbps without any issues with Ceph running.
Groovy. These are all things we have to ask ;)
Check the interfaces to ensure they have the proper netmasks and default routes;
I found some systems with the main interface configured as a /32 it's top of mind lately.
They do.
Thanks!