Hello, last week I've got a HEALTH_OK on our CEPH cluster and I started upgrade firmware in network cards. When I had upgraded the sixth card from nine (one-by-one), this server didn't started correctly and our ProxMox had problem with accessing disk images on CEPH. rbd ls pool was OK, but: rbd ls pool -l didn't work. Our virtual servers had a trouble to work with disks. After I resolve network problem with OSD server, everythink returning to normal state. But I've found, that every OSD nod have very high activity: when I've started 'iotop', there was very high load: around 180MB/s read and 20MB/s write. In this time, cluster was in the HEALTH_OK state. I've found, that there is a massive scrubbing activity... After a few days, I have on our OSD nodes around 90MB/s read and 70MB/s write while 'ceph -s' have client io as 2,5MB/s read and 50MB/s write. I've found in log file of our mon server many lines about starting of scrubbing, but there are many messages about starting of scrubb the same PG? I've grep'ed syslog for some of them and attach it to this e-mail. Is this activity OK? Why CEPH start scrubing this PG once and once again? And another question: Is scrubbing part of mClock scheduler? Many thanks for explanation. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html