David G, I've been looking over logs and ceph pg dump pgs on the LRC and things look good to me. If you see anything not working file a tracker or if you have any questions please contact me. There is one thing that you should be aware of. There are still filestore objectstores for some of the OSDs. The auto_repair feature is not supported for filestore, so when they deep-scrub they won't repair. With auto_repair enabled in this mixed cluster the LRC will auto_repair if the primary OSD for a PG is bluestore even if some replicas are filestore. So I would convert the the remaining filestore OSDs to bluestore. If you are paranoid you should disable auto_repair until the conversion is completed. David Z On 7/2/19 3:11 PM, David Zafman wrote:
I don't see that now in ceph status. A pg's deep scrub would have to be over 5 days overdue for that warning to occur.
David
On 7/2/19 2:29 PM, David Galloway wrote:
This build is installed now.
It looks like "1 pgs not scrubbed in time" is back.
On 6/28/19 12:27 PM, David Zafman wrote:
David,
I have a new scrub handling code built for Nautilus. Could we install this on the LRC to see how well it works in a more realistic environment?
https://shaman.ceph.com/builds/ceph/wip-zafman-testing-nautilus/31ff31f2c8d3...
Thanks
David Zafman
Thanks David. I'll work on converting the rest of the OSDs to bluestore. In the meantime, it looks like all the mgrs died which is new I think? 2019-07-08 14:00:12.022 7f4733140700 -1 received signal: Terminated from /sbin/init splash (PID: 1) UID: 0 2019-07-08 14:00:12.022 7f4733140700 -1 mgr handle_signal *** Got signal Terminated *** Can you take a look please? On 7/3/19 7:09 PM, David Zafman wrote:
David G,
I've been looking over logs and ceph pg dump pgs on the LRC and things look good to me. If you see anything not working file a tracker or if you have any questions please contact me.
There is one thing that you should be aware of. There are still filestore objectstores for some of the OSDs. The auto_repair feature is not supported for filestore, so when they deep-scrub they won't repair. With auto_repair enabled in this mixed cluster the LRC will auto_repair if the primary OSD for a PG is bluestore even if some replicas are filestore. So I would convert the the remaining filestore OSDs to bluestore. If you are paranoid you should disable auto_repair until the conversion is completed.
David Z
On 7/2/19 3:11 PM, David Zafman wrote:
I don't see that now in ceph status. A pg's deep scrub would have to be over 5 days overdue for that warning to occur.
David
On 7/2/19 2:29 PM, David Galloway wrote:
This build is installed now.
It looks like "1 pgs not scrubbed in time" is back.
On 6/28/19 12:27 PM, David Zafman wrote:
David,
I have a new scrub handling code built for Nautilus. Could we install this on the LRC to see how well it works in a more realistic environment?
https://shaman.ceph.com/builds/ceph/wip-zafman-testing-nautilus/31ff31f2c8d3...
Thanks
David Zafman
I guess I should check first... @Sage, are you okay with converting all the remaining Sepia LRC filestore OSDs to bluestore? I just remembered you wanted to keep a mixture a couple years ago but maybe that's changed. On 7/10/19 11:14 AM, David Galloway wrote:
Thanks David.
I'll work on converting the rest of the OSDs to bluestore.
In the meantime, it looks like all the mgrs died which is new I think?
2019-07-08 14:00:12.022 7f4733140700 -1 received signal: Terminated from /sbin/init splash (PID: 1) UID: 0 2019-07-08 14:00:12.022 7f4733140700 -1 mgr handle_signal *** Got signal Terminated ***
Can you take a look please?
On 7/3/19 7:09 PM, David Zafman wrote:
David G,
I've been looking over logs and ceph pg dump pgs on the LRC and things look good to me. If you see anything not working file a tracker or if you have any questions please contact me.
There is one thing that you should be aware of. There are still filestore objectstores for some of the OSDs. The auto_repair feature is not supported for filestore, so when they deep-scrub they won't repair. With auto_repair enabled in this mixed cluster the LRC will auto_repair if the primary OSD for a PG is bluestore even if some replicas are filestore. So I would convert the the remaining filestore OSDs to bluestore. If you are paranoid you should disable auto_repair until the conversion is completed.
David Z
On 7/2/19 3:11 PM, David Zafman wrote:
I don't see that now in ceph status. A pg's deep scrub would have to be over 5 days overdue for that warning to occur.
David
On 7/2/19 2:29 PM, David Galloway wrote:
This build is installed now.
It looks like "1 pgs not scrubbed in time" is back.
On 6/28/19 12:27 PM, David Zafman wrote:
David,
I have a new scrub handling code built for Nautilus. Could we install this on the LRC to see how well it works in a more realistic environment?
https://shaman.ceph.com/builds/ceph/wip-zafman-testing-nautilus/31ff31f2c8d3...
Thanks
David Zafman
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
On Thu, 11 Jul 2019, David Galloway wrote:
I guess I should check first...
@Sage, are you okay with converting all the remaining Sepia LRC filestore OSDs to bluestore? I just remembered you wanted to keep a mixture a couple years ago but maybe that's changed.
Yeah, let's do it! Note that I have one piece of maint I want to do first, though.. see https://github.com/ceph/ceph/pull/28978. I want to build the natuilus backport of this and deploy it and make sure it behaves better. Hopefully I'll have that tested by tomorrow. s
On 7/10/19 11:14 AM, David Galloway wrote:
Thanks David.
I'll work on converting the rest of the OSDs to bluestore.
In the meantime, it looks like all the mgrs died which is new I think?
2019-07-08 14:00:12.022 7f4733140700 -1 received signal: Terminated from /sbin/init splash (PID: 1) UID: 0 2019-07-08 14:00:12.022 7f4733140700 -1 mgr handle_signal *** Got signal Terminated ***
Can you take a look please?
On 7/3/19 7:09 PM, David Zafman wrote:
David G,
I've been looking over logs and ceph pg dump pgs on the LRC and things look good to me. If you see anything not working file a tracker or if you have any questions please contact me.
There is one thing that you should be aware of. There are still filestore objectstores for some of the OSDs. The auto_repair feature is not supported for filestore, so when they deep-scrub they won't repair. With auto_repair enabled in this mixed cluster the LRC will auto_repair if the primary OSD for a PG is bluestore even if some replicas are filestore. So I would convert the the remaining filestore OSDs to bluestore. If you are paranoid you should disable auto_repair until the conversion is completed.
David Z
On 7/2/19 3:11 PM, David Zafman wrote:
I don't see that now in ceph status. A pg's deep scrub would have to be over 5 days overdue for that warning to occur.
David
On 7/2/19 2:29 PM, David Galloway wrote:
This build is installed now.
It looks like "1 pgs not scrubbed in time" is back.
On 6/28/19 12:27 PM, David Zafman wrote:
David,
I have a new scrub handling code built for Nautilus. Could we install this on the LRC to see how well it works in a more realistic environment?
https://shaman.ceph.com/builds/ceph/wip-zafman-testing-nautilus/31ff31f2c8d3...
Thanks
David Zafman
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
_______________________________________________ Sepia mailing list -- sepia@ceph.io To unsubscribe send an email to sepia-leave@ceph.io
participants (3)
-
David Galloway
-
David Zafman
-
Sage Weil