"cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"
27 Oct
2023
27 Oct
'23
5:48 a.m.
Hi
I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see
the helpful replies because my mail service binned the responses.
I still can't get the reef version of cephadm to work properly.
I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph
daemons to reef before my first email.
Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.*
return the same error when running "cephadm version"
Traceback (most recent call last):
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 9468, in <module>
main()
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 9456, in main
r = ctx.func(ctx)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2108, in _infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2191, in infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not None)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2154, in get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and
d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2154, in <listcomp>
matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and
d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 217, in __getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'
I am running into other issues as well, but I think they may point back to this issue of
"'CephadmContext' object has no attribute 'fsid'"
Any help would be appreciated.
Regards,
Martin Conway
IT and Digital Media Manager
Research School of Physics
Australian National University
Canberra ACT 2601
+61 2 6125 1599
https://physics.anu.edu.au<https://physics.anu.edu.au/>
27 Oct
27 Oct
5:53 a.m.
I just had another look through the issues tracker and found this bug already listed.
https://tracker.ceph.com/issues/59428
I need to go back to the other issues I am having and figure out if they are related or
something different.
Hi
I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see
the helpful replies because my mail service binned the responses.
I still can't get the reef version of cephadm to work properly.
I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph
daemons to reef before my first email.
Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.*
return the same error when running "cephadm version"
Traceback (most recent call last):
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 9468, in <module>
main()
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 9456, in main
r = ctx.func(ctx)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2108, in _infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2191, in infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not None)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2154, in get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and
d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2154, in <listcomp>
matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and
d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 217, in __getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'
9:40 a.m.
Are the issues you refer to the same as before? I don't think this
version issue is the root cause, I do see it as well in my test
cluster(s) but the rest works properly except for the tag issue I
already reported which you can easily fix by setting the config value
for the default image
(https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGD…).
Or are there new issues you
encountered?
Zitat von Martin Conway <martin.conway(a)anu.edu.au>au>:
I just had another look through the issues tracker and
found this
bug already listed.
https://tracker.ceph.com/issues/59428
I need to go back to the other issues I am having and figure out if
they are related or something different.
Hi
I wrote before about issues I was having with cephadm in 18.2.0
Sorry, I didn't see the helpful replies because my mail service
binned the responses.
I still can't get the reef version of cephadm to work properly.
I had updated the system rpm to reef (ceph repo) and also upgraded
the containerised ceph daemons to reef before my first email.
Both the system package cephadm and the one found at
/var/lib/ceph/${fsid}/cephadm.* return the same error when running
"cephadm version"
Traceback (most recent call last):
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 9468, in
<module>
main()
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 9456, in
main
r = ctx.func(ctx)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2108, in
_infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2191, in
infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not None)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2154, in
get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 2154, in
<listcomp>
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e",
line 217, in
__getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:57 p.m.
On Friday, October 27, 2023 2:40:17 AM EDT Eugen Block wrote:
Are the issues you refer to the same as before? I
don't think this
version issue is the root cause, I do see it as well in my test
cluster(s) but the rest works properly except for the tag issue I
already reported which you can easily fix by setting the config value
for the default image
(https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGD
YAWPVE2YLV2ZLF3HC5SLS/#LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS). Or are there new
issues you encountered?
I concur. That `cephadm version` failure is/was a known issue but should not
be the cause of any other issues. On the main branch `cephadm version` no
longer fails this way - rather, it reports the version of a cephadm build and
no longer inspects a container image. We can look into backporting this
before the next reef release.
The issue related to the container image tag that Eugen filed has also been
fixed on reef. Thanks for filing that.
Martin you may want to retry things after the next reef release.
Unfortunately, I don't know when that is planned but I think it's soonish.
Zitat von Martin Conway <martin.conway(a)anu.edu.au>au>:
I just had another look through the issues
tracker and found this
bug already listed.
https://tracker.ceph.com/issues/59428
I need to go back to the other issues I am having and figure out if
they are related or something different.
Hi
I wrote before about issues I was having with cephadm in 18.2.0
Sorry, I didn't see the helpful replies because my mail service
binned the responses.
I still can't get the reef version of cephadm to work properly.
I had updated the system rpm to reef (ceph repo) and also upgraded
the containerised ceph daemons to reef before my first email.
Both the system package cephadm and the one found at
/var/lib/ceph/${fsid}/cephadm.* return the same error when running
"cephadm version"
Traceback (most recent call last):
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4
e", line 9468, in <module>
main()
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4
e", line 9456, in main
r = ctx.func(ctx)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4
e", line 2108, in _infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4
e", line 2191, in infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not
None)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4
e", line 2154, in get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4
e", line 2154, in <listcomp>
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4
e", line 217, in __getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1 Nov
1 Nov
12:35 a.m.
Sorry for the delayed response. Office365 keeps quarantining 75% of the emails from this
list.
I had a couple of other issues, one of which seems to have come good by itself.
cephadm adopt --style legacy --name osd.12
Would fail, but I tried it again yesterday without having made any other changes I can
think of, and it worked fine. Sorry, I don't have a record of what the error message
was.
I have been trying to add SSD DB/WAL to all of my Bluestor OSD spinning disks. The
commands I found to simply move the DB didn't work for me so I have been removing each
OSD and recreating again using ceph-colume, then adopting again. This process has been
going on for weeks, and I now have a persistent issue with scrubs not having been
completed in time. I am unsure if this will sort itself out if I just leave it alone for a
week or two, or if there is an underlying issue with Reef and scrubbing being slow. It
definitely doesn't fix itself if left alone for a day or two.
I find that backfilling and possibly scrubbing often comes to a halt for no apparent
reason. If I put a server into maintenance mode or kill and restart OSDs it bursts back
into life again.
Not sure how to diagnose why the recovery processes have stalled.
Regards,
Martin
> -----Original Message-----
> From: John Mulligan <phlogistonjohn(a)asynchrono.us>
> Sent: Saturday, October 28, 2023 12:58 AM
> To: ceph-users(a)ceph.io
> Subject: [ceph-users] Re: "cephadm version" in reef returns
"AttributeError:
> 'CephadmContext' object has no attribute 'fsid'"
>
> [You don't often get email from phlogistonjohn(a)asynchrono.us. Learn why
> this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> On Friday, October 27, 2023 2:40:17 AM EDT Eugen Block wrote:
> > Are the issues you refer to the same as before? I don't think this
> > version issue is the root cause, I do see it as well in my test
> > cluster(s) but the rest works properly except for the tag issue I
> > already reported which you can easily fix by setting the config value
> > for the default image
> > (https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
> > ts.ceph.io%2Fhyperkitty%2Flist%2Fceph-
> users%40ceph.io%2Fthread%2FLASBJ
> >
> CSPFGD&data=05%7C01%7Cmartin.conway%40anu.edu.au%7Cbc11333aca
> 854f478a0
> >
> 308dbd6f4d47e%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C
> 63834310466
> >
> 8081112%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
> V2luMzIiLC
> >
> JBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iaeVqSYY8
> %2B6a8mqZ1
> > hgGmGif5%2BAjrWgXqeuhGaRKy7Q%3D&reserved=0
> > YAWPVE2YLV2ZLF3HC5SLS/#LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS). Or
> are there
> > new issues you encountered?
>
>
> I concur. That `cephadm version` failure is/was a known issue but should not
> be the cause of any other issues. On the main branch `cephadm version` no
> longer fails this way - rather, it reports the version of a cephadm build and no
> longer inspects a container image. We can look into backporting this before
> the next reef release.
>
> The issue related to the container image tag that Eugen filed has also been
> fixed on reef. Thanks for filing that.
>
> Martin you may want to retry things after the next reef release.
> Unfortunately, I don't know when that is planned but I think it's soonish.
>
> >
> > Zitat von Martin Conway <martin.conway(a)anu.edu.au>au>:
> > > I just had another look through the issues tracker and found this
> > > bug already listed.
> > > https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftr
> > >
> acker.ceph.com%2Fissues%2F59428&data=05%7C01%7Cmartin.conway%40
> anu.e
> > >
> du.au%7Cbc11333aca854f478a0308dbd6f4d47e%7Ce37d725cab5c46249a
> e5f0533
> > >
> e486437%7C0%7C0%7C638343104668081112%7CUnknown%7CTWFpbGZ
> sb3d8eyJWIjo
> > >
> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
> 000%7
> > >
> C%7C%7C&sdata=kYRi9R4hSa5CPErQEct5MRUftPG%2FpFcLHu30%2FfpWUQ
> w%3D&res
> > > erved=0
> > >
> > > I need to go back to the other issues I am having and figure out if
> > > they are related or something different.
> > >
> > >
> > > Hi
> > >
> > > I wrote before about issues I was having with cephadm in 18.2.0
> > > Sorry, I didn't see the helpful replies because my mail service
> > > binned the responses.
> > >
> > > I still can't get the reef version of cephadm to work properly.
> > >
> > > I had updated the system rpm to reef (ceph repo) and also upgraded
> > > the containerised ceph daemons to reef before my first email.
> > >
> > > Both the system package cephadm and the one found at
> > > /var/lib/ceph/${fsid}/cephadm.* return the same error when running
> > > "cephadm version"
> > >
> > > Traceback (most recent call last):
> > > File
> > >
> > >
> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
> 05c
> > > afd9b4
> > > e", line 9468, in <module>
> > >
> > > main()
> > >
> > > File
> > >
> > >
> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
> 05c
> > > afd9b4
> > > e", line 9456, in main
> > >
> > > r = ctx.func(ctx)
> > >
> > > File
> > >
> > >
> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
> 05c
> > > afd9b4
> > > e", line 2108, in _infer_image
> > >
> > > ctx.image = infer_local_ceph_image(ctx,
> > > ctx.container_engine.path)
> > >
> > > File
> > >
> > >
> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
> 05c
> > > afd9b4 e", line 2191, in infer_local_ceph_image
> > >
> > > container_info = get_container_info(ctx, daemon, daemon_name is not
> > > None)
> > >
> > > File
> > >
> > >
> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
> 05c
> > > afd9b4 e", line 2154, in get_container_info
> > >
> > > matching_daemons = [d for d in daemons if daemon_name_or_type(d)
> > >
> > > == daemon_filter and d['fsid'] == ctx.fsid]
> > >
> > > File
> > >
> > >
> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
> 05c
> > > afd9b4
> > > e", line 2154, in <listcomp>
> > >
> > > matching_daemons = [d for d in daemons if daemon_name_or_type(d)
> > >
> > > == daemon_filter and d['fsid'] == ctx.fsid]
> > >
> > > File
> > >
> > >
> "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
> 05c
> > > afd9b4
> > > e", line 217, in __getattr__
> > >
> > > return super().__getattribute__(name)
> > >
> > > AttributeError: 'CephadmContext' object has no attribute
'fsid'
> > >
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> > > email to ceph-users-leave(a)ceph.io
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> > email to ceph-users-leave(a)ceph.io
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
> ceph-users-leave(a)ceph.io
12:52 a.m.
Martin Conway wrote:
I find that backfilling and possibly scrubbing often
comes to a halt for no apparent
reason. If I put a server into maintenance mode or kill and restart OSDs it bursts back
into life again.
Not sure how to diagnose why the recovery processes have stalled.
My cluster is in this stalled state now, I have saved some details below.
Seems to point quite heavily to OSD.32 and OSD.33 but there is nothing of note in their
logs. They were experiencing slow ops last night, and this morning have logged nothing. I
am certain recovery and scrubbing will resume if I restarted those OSDs, but it would be
nice to know what keep causing this.
ceph -s
cluster:
id: 16bb4f7a-cf04-4667-aeee-94ce7f6ab672
health: HEALTH_WARN
441 pgs not deep-scrubbed in time
43 pgs not scrubbed in time
services:
mon: 5 daemons, quorum scustor3,scustor2,scustor1,scustor4,scustor5 (age 23h)
mgr: scustor3.wplaov(active, since 2d), standbys: scustor4.giyegr, scustor1.luywbi,
scustor2.ncfaec
mds: 2/2 daemons up, 1 standby
osd: 31 osds: 31 up (since 23h), 31 in (since 4d); 54 remapped pgs
data:
volumes: 1/1 healthy
pools: 8 pools, 897 pgs
objects: 48.52M objects, 47 TiB
usage: 130 TiB used, 130 TiB / 260 TiB avail
pgs: 3497437/145569867 objects misplaced (2.403%)
757 active+clean
70 active+clean+scrubbing
48 active+remapped+backfilling
14 active+clean+scrubbing+deep
6 active+recovering+remapped
2 active+recovering
io:
client: 24 KiB/s rd, 416 KiB/s wr, 1 op/s rd, 44 op/s wr
ceph pg dump
https://pastebin.com/raw/KPBie7SD
ceph pg dump_stuck
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY
5.1f4 active+remapped+backfilling [32,5,18] 32 [32,5,21] 32
5.1f0 active+remapped+backfilling [32,2,13] 32 [32,2,5] 32
5.1e4 active+remapped+backfilling [33,13,10] 33 [33,6,5] 33
5.1c5 active+remapped+backfilling [33,2,10] 33 [33,10,16] 33
5.1bc active+remapped+backfilling [18,33,11] 18 [33,11,6] 33
5.193 active+remapped+backfilling [33,14,3] 33 [33,13,5] 33
5.180 active+remapped+backfilling [32,13,1] 32 [32,1,10] 32
5.171 active+remapped+backfilling [33,1,18] 33 [33,1,20] 33
5.16b active+recovering+remapped [32,9,13] 32 [32,9,2] 32
5.16a active+remapped+backfilling [13,4,6] 13 [33,4,16] 33
5.169 active+remapped+backfilling [14,33,10] 14 [33,10,13] 33
5.162 active+remapped+backfilling [32,6,13] 32 [32,6,22] 32
5.130 active+remapped+backfilling [13,32,3] 13 [32,3,1] 32
5.1cd active+remapped+backfilling [33,5,18] 33 [33,5,13] 33
6.3b active+recovering [32,1,3] 32 [32,1,3] 32
6.42 active+remapped+backfilling [18,33,9] 18 [33,9,13] 33
5.4e active+remapped+backfilling [32,11,10] 32 [32,10,22] 32
5.167 active+remapped+backfilling [33,18,6] 33 [33,6,2] 33
6.20 active+remapped+backfilling [14,32,10] 14 [32,13,1] 32
5.52 active+remapped+backfilling [14,1,32] 14 [32,1,13] 32
5.57 active+recovering+remapped [32,9,13] 32 [32,9,22] 32
5.49 active+remapped+backfilling [18,32,9] 18 [32,9,13] 32
5.1f7 active+recovering+remapped [9,32,13] 9 [32,20,1] 32
5.100 active+remapped+backfilling [33,9,13] 33 [33,9,20] 33
5.58 active+remapped+backfilling [32,6,4] 32 [32,6,20] 32
5.16 active+recovering+remapped [32,18,3] 32 [32,3,9] 32
5.60 active+recovering+remapped [31,13,4] 31 [32,4,9] 32
5.fc active+recovering [32,9,10] 32 [32,9,10] 32
5.c8 active+remapped+backfilling [32,14,3] 32 [32,3,13] 32
5.ad active+remapped+backfilling [32,3,16] 32 [32,3,22] 32
5.6e active+remapped+backfilling [32,5,18] 32 [32,5,11] 32
6.6d active+remapped+backfilling [32,6,5] 32 [32,5,21] 32
5.cf active+remapped+backfilling [33,9,18] 33 [33,11,16] 33
5.7e active+remapped+backfilling [32,9,18] 32 [32,4,16] 32
6.37 active+remapped+backfilling [32,4,3] 32 [32,4,21] 32
5.1aa active+remapped+backfilling [13,33,10] 13 [33,10,1] 33
5.165 active+remapped+backfilling [33,5,16] 33 [33,5,21] 33
5.76 active+remapped+backfilling [13,1,32] 13 [32,1,22] 32
5.102 active+remapped+backfilling [33,5,6] 33 [33,5,21] 33
5.2d active+remapped+backfilling [32,18,4] 32 [32,4,5] 32
6.24 active+remapped+backfilling [33,18,2] 33 [33,9,3] 33
5.f6 active+remapped+backfilling [32,1,14] 32 [32,1,22] 32
5.1c active+remapped+backfilling [33,18,3] 33 [33,3,22] 33
5.d9 active+remapped+backfilling [33,18,11] 33 [33,11,9] 33
5.184 active+remapped+backfilling [32,14,2] 32 [32,20,5] 32
5.e6 active+remapped+backfilling [18,33,16] 18 [33,16,13] 33
5.18f active+recovering+remapped [18,32,9] 18 [32,9,13] 32
5.e9 active+remapped+backfilling [32,13,9] 32 [32,13,2] 32
5.55 active+remapped+backfilling [32,6,14] 32 [32,6,3] 32
5.eb active+remapped+backfilling [18,33,11] 18 [32,20,6] 32
6.13 active+remapped+backfilling [14,10,1] 14 [32,20,13] 32
5.107 active+remapped+backfilling [14,3,31] 14 [32,3,1] 32
5.109 active+remapped+backfilling [32,4,14] 32 [32,4,13] 32
5.117 active+remapped+backfilling [33,16,3] 33 [33,16,20] 33
6.30 active+remapped+backfilling [32,4,1] 32 [32,1,21] 32
5.126 active+remapped+backfilling [33,9,4] 33 [33,9,21] 33
ok
7:02 p.m.
Hi,
first of all, I'd still recommend to use the orchestrator to deploy
OSDs. Building OSDs manually and then adopt them is redundant. Or do
you have issues with the drivegroups?
I don't have *the* solution but you could try to disable the mclock
scheduler [1] which is the default since Quincy. Maybe that will speed
up things? There have been reports in the list about some unwanted or
at least unexpected behavior.
As for the "not (deep-)scrubbed in time" messages, there seems to be
progress (in your ceph status), but depending on the drive utilization
you could increase the number of scrubs per OSD (osd_max_scrubs).
Regards,
Eugen
[1] https://www.clyso.com/blog/ceph-how-do-disable-mclock-scheduler/
Zitat von Martin Conway <martin.conway(a)anu.edu.au>au>:
Sorry for the delayed response. Office365 keeps
quarantining 75% of
the emails from this list.
I had a couple of other issues, one of which seems to have come good
by itself.
cephadm adopt --style legacy --name osd.12
Would fail, but I tried it again yesterday without having made any
other changes I can think of, and it worked fine. Sorry, I don't
have a record of what the error message was.
I have been trying to add SSD DB/WAL to all of my Bluestor OSD
spinning disks. The commands I found to simply move the DB didn't
work for me so I have been removing each OSD and recreating again
using ceph-colume, then adopting again. This process has been going
on for weeks, and I now have a persistent issue with scrubs not
having been completed in time. I am unsure if this will sort itself
out if I just leave it alone for a week or two, or if there is an
underlying issue with Reef and scrubbing being slow. It definitely
doesn't fix itself if left alone for a day or two.
I find that backfilling and possibly scrubbing often comes to a halt
for no apparent reason. If I put a server into maintenance mode or
kill and restart OSDs it bursts back into life again.
Not sure how to diagnose why the recovery processes have stalled.
Regards,
Martin
-----Original Message-----
From: John Mulligan <phlogistonjohn(a)asynchrono.us>
Sent: Saturday, October 28, 2023 12:58 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: "cephadm version" in reef returns
"AttributeError:
'CephadmContext' object has no attribute 'fsid'"
[You don't often get email from phlogistonjohn(a)asynchrono.us. Learn why
this is important at https://aka.ms/LearnAboutSenderIdentification ]
On Friday, October 27, 2023 2:40:17 AM EDT Eugen Block wrote:
CSPFGD&data=05%7C01%7Cmartin.conway%40anu.edu.au%7Cbc11333aca
854f478a0
308dbd6f4d47e%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C
63834310466
8081112%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
V2luMzIiLC
JBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iaeVqSYY8
%2B6a8mqZ1
> Zitat von Martin Conway
<martin.conway(a)anu.edu.au>au>:
> > I just had another look through the issues tracker and found this
> > bug already listed.
> > https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftr
>
acker.ceph.com%2Fissues%2F59428&data=05%7C01%7Cmartin.conway%40
anu.e
>
du.au%7Cbc11333aca854f478a0308dbd6f4d47e%7Ce37d725cab5c46249a
e5f0533
>
e486437%7C0%7C0%7C638343104668081112%7CUnknown%7CTWFpbGZ
sb3d8eyJWIjo
>
iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
000%7
>
C%7C%7C&sdata=kYRi9R4hSa5CPErQEct5MRUftPG%2FpFcLHu30%2FfpWUQ
w%3D&res
> > erved=0
> > > I need to go back to the
other issues I am having and figure out if
> > they are related or something different.
> > > > Hi
> > > I wrote before about issues
I was having with cephadm in 18.2.0
> > Sorry, I didn't see the helpful replies because my mail service
> > binned the responses.
> > > I still can't get the
reef version of cephadm to work properly.
> > > I had updated the system
rpm to reef (ceph repo) and also upgraded
> > the containerised ceph daemons to reef before my first email.
> > > Both the system package
cephadm and the one found at
> > /var/lib/ceph/${fsid}/cephadm.* return the same error when running
> > "cephadm version"
> > > Traceback (most recent call
last):
> > File
> >
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
05c
> > afd9b4
> > e", line 9468, in <module>
> > > main()
> > > File
> >
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
05c
> > afd9b4
> > e", line 9456, in main
> > > r = ctx.func(ctx)
> > > File
> >
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
05c
> > afd9b4
> > e", line 2108, in _infer_image
> > > ctx.image =
infer_local_ceph_image(ctx,
> > ctx.container_engine.path)
> > > File
> >
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
05c
> > afd9b4 e", line 2191, in infer_local_ceph_image
> > > container_info =
get_container_info(ctx, daemon, daemon_name is not
> > None)
> > > File
> >
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
05c
> > afd9b4 e", line 2154, in get_container_info
> > > matching_daemons = [d
for d in daemons if daemon_name_or_type(d)
> > > == daemon_filter and
d['fsid'] == ctx.fsid]
> > > File
> >
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
05c
> > afd9b4
> > e", line 2154, in <listcomp>
> > > matching_daemons = [d
for d in daemons if daemon_name_or_type(d)
> > > == daemon_filter and
d['fsid'] == ctx.fsid]
> > > File
> >
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1
05c
> > afd9b4
> > e", line 217, in __getattr__
> > > return
super().__getattribute__(name)
> > > AttributeError:
'CephadmContext' object has no attribute 'fsid'
> > >
_______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> > email to ceph-users-leave(a)ceph.io
>
_______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send
an email to
ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Are the issues you refer to the same as before? I
don't think this
version issue is the root cause, I do see it as well in my test
cluster(s) but the rest works properly except for the tag issue I
already reported which you can easily fix by setting the config value
for the default image
(https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
ts.ceph.io%2Fhyperkitty%2Flist%2Fceph-
users%40ceph.io%2Fthread%2FLASBJ
hgGmGif5%2BAjrWgXqeuhGaRKy7Q%3D&reserved=0
YAWPVE2YLV2ZLF3HC5SLS/#LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS). Or
are there
new issues you encountered?
I concur. That `cephadm version` failure is/was a known issue but should not
be the cause of any other issues. On the main branch `cephadm version` no
longer fails this way - rather, it reports the version of a cephadm
build and no
longer inspects a container image. We can look into backporting this before
the next reef release.
The issue related to the container image tag that Eugen filed has also been
fixed on reef. Thanks for filing that.
Martin you may want to retry things after the next reef release.
Unfortunately, I don't know when that is planned but I think it's soonish.
2 Nov
2 Nov
12:56 a.m.
first of all, I'd still recommend to use the
orchestrator to deploy OSDs. Building
OSDs manually and then adopt them is redundant. Or do you have issues with
the drivegroups?
I am having to do it this way because I couldn't find any doco on how to specify a
separate DB/WAL device when deploying OSDs using the orchestrator. If there is such a
command I agree it would be a better choice.
Ideally I would use the commands to simply move the DB of my existing orchestrator
deployed ODSs to the SSD, but when I tried that command it broke my OSD and I had to
delete it and leave he cluster in a degraded state until it had recovered. I find it very
stressful when I get out of my depth with problems like that, so I gave up on that idea
and am doing the remove, redploy, adopt method, which is working, but VERY slow.
I don't have *the* solution but you could try to
disable the mclock scheduler
[1] which is the default since Quincy. Maybe that will speed up things? There
have been reports in the list about some unwanted or at least unexpected
behavior.
I did try this to try and speed up my rebalances, but it didn't seem to make much
difference. I haven't tried it to see what difference it makes to scrubbing.
As for the "not (deep-)scrubbed in time"
messages, there seems to be
progress (in your ceph status), but depending on the drive utilization you
could increase the number of scrubs per OSD (osd_max_scrubs).
There are lot of scrubs running, this morning after my rebalance finally completed it has
22 scrubbing, 6 deep scrubbing (across 28 OSDs). This has fallen from the number it was
running yesterday when the rebalance was still happening (38/9).
I believe if I kick the cluster by taking a host into maintenance and back the numbers
will jump up again. The trouble is I don't know how to tell if a scrub is actually
achieving something, stuck or restarting over and over.
My current ceph pg dump is:
https://pastebin.com/AQhNKSBN
and if I run it again a few minutes later:
https://pastebin.com/yfREzJ4s
I see evidence of scrubs not working because some of my OSD logs look like this:
2023-11-01T20:51:08.668+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub
starts
2023-11-01T20:51:11.658+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub
starts
2023-11-01T20:51:19.565+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub
starts
2023-11-01T20:51:20.516+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub
starts
2023-11-01T20:51:22.463+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub
starts
2023-11-01T20:51:24.488+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub
starts
2023-11-01T20:51:29.474+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub
starts
2023-11-01T20:51:31.484+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub
starts
2023-11-01T20:51:34.455+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub
starts
2023-11-01T20:51:39.444+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9
deep-scrub starts
2023-11-01T20:51:42.473+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub
starts
2023-11-01T20:51:44.510+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub
starts
2023-11-01T20:51:46.491+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub
starts
2023-11-01T20:51:47.465+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub
starts
2023-11-01T20:51:49.443+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub
starts
2023-11-01T20:51:51.439+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub
starts
2023-11-01T20:51:53.388+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub
starts
2023-11-01T20:52:00.345+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub
starts
2023-11-01T20:52:02.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub
starts
2023-11-01T20:52:03.452+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub
starts
2023-11-01T20:52:10.421+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub
starts
2023-11-01T20:52:11.436+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9
deep-scrub starts
2023-11-01T20:52:12.465+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b
deep-scrub starts
2023-11-01T20:52:13.470+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub
starts
2023-11-01T20:52:14.468+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d
deep-scrub starts
2023-11-01T20:52:17.512+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub
starts
2023-11-01T20:52:20.507+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub
starts
2023-11-01T20:52:22.428+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub
starts
2023-11-01T20:52:23.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub
starts
2023-11-01T20:52:24.444+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub
starts
2023-11-01T20:52:28.461+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub
starts
2023-11-01T20:52:45.551+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub
starts
2023-11-01T20:52:46.593+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub
starts
2023-11-01T20:52:48.595+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub
starts
2023-11-01T20:52:52.488+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b
deep-scrub starts
2023-11-01T20:52:55.504+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub
starts
2023-11-01T20:52:58.519+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub
starts
2023-11-01T20:52:59.477+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub
starts
2023-11-01T20:53:01.505+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub
starts
2023-11-01T20:53:03.467+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub
starts
2023-11-01T20:53:06.406+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub
starts
2023-11-01T20:53:10.446+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub
starts
2023-11-01T20:53:13.470+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub
starts
2023-11-01T20:53:15.521+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub
starts
There is no scrub OK type messages.
I notice these logs were from yesterday, and currently there is nothing being logged for
these OSDs. If I was to "kick the cluster" these scrub logs would probably start
showing up again.
When this is happening the PGs mentioned seem to stay in a "scrub queued"
state.
Any light you could shine my way would be appreciated.
Thanks,
Martin
10:52 a.m.
There are a couple of examples in the docs [2], so in your case it
probably would be something rather simple like:
service_type: osd
service_id: osd_spec_default
placement:
host_pattern: '*'
spec:
data_devices:
rotational: 1
db_devices:
rotational: 0
You can apply that config to specific hosts or all of them, it really
depends on your actual setup. You can also dry-run the config before
applying it with the --dry-run flag:
ceph orch apply -i my-osd-specs.yaml --dry-run
I'd recommend to create a test cluster if possible to have some
options to practice and get familiar with all that stuff.
Ideally I would use the commands to simply move the DB
of my
existing orchestrator deployed ODSs to the SSD, but when I tried
that command it broke my OSD and I had to delete it and leave he
cluster in a degraded state until it had recovered.
Do you still have the commands and the output somewhere what exactly
went wrong? I haven't migrated DBs in quite some time, especially not
in a newer version like Reef. I assume you tried it with
bluefs-bdev-migrate?
I remember having seen these repeating scrub starts messages in this
list, but I can't seem to find the right thread. I can't recall if
there was a solution to that...
When was the last time you failed the mgr service? That still does
help sometimes...
[2] https://docs.ceph.com/en/reef/cephadm/services/osd/#examples
Zitat von Martin Conway <martin.conway(a)anu.edu.au>au>:
first of all,
I'd still recommend to use the orchestrator to deploy
OSDs. Building
OSDs manually and then adopt them is redundant. Or do you have issues with
the drivegroups?
I am having to do it this way because I couldn't find any doco on
how to specify a separate DB/WAL device when deploying OSDs using
the orchestrator. If there is such a command I agree it would be a
better choice.
Ideally I would use the commands to simply move the DB of my
existing orchestrator deployed ODSs to the SSD, but when I tried
that command it broke my OSD and I had to delete it and leave he
cluster in a degraded state until it had recovered. I find it very
stressful when I get out of my depth with problems like that, so I
gave up on that idea and am doing the remove, redploy, adopt method,
which is working, but VERY slow.
I don't have *the* solution but you could try
to disable the mclock
scheduler
[1] which is the default since Quincy. Maybe that will speed up
things? There
have been reports in the list about some unwanted or at least unexpected
behavior.
I did try this to try and speed up my rebalances, but it didn't seem
to make much difference. I haven't tried it to see what difference
it makes to scrubbing.
As for the "not (deep-)scrubbed in
time" messages, there seems to be
progress (in your ceph status), but depending on the drive utilization you
could increase the number of scrubs per OSD (osd_max_scrubs).
There are lot of scrubs running, this morning after my rebalance
finally completed it has 22 scrubbing, 6 deep scrubbing (across 28
OSDs). This has fallen from the number it was running yesterday when
the rebalance was still happening (38/9).
I believe if I kick the cluster by taking a host into maintenance
and back the numbers will jump up again. The trouble is I don't know
how to tell if a scrub is actually achieving something, stuck or
restarting over and over.
My current ceph pg dump is:
https://pastebin.com/AQhNKSBN
and if I run it again a few minutes later:
https://pastebin.com/yfREzJ4s
I see evidence of scrubs not working because some of my OSD logs
look like this:
2023-11-01T20:51:08.668+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 6.2d scrub starts
2023-11-01T20:51:11.658+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1ac scrub starts
2023-11-01T20:51:19.565+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.17 scrub starts
2023-11-01T20:51:20.516+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1d9 scrub starts
2023-11-01T20:51:22.463+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 5.9b scrub starts
2023-11-01T20:51:24.488+0000 7f3be0b27700 0 log_channel(cluster)
log [DBG] : 5.65 scrub starts
2023-11-01T20:51:29.474+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 6.2d scrub starts
2023-11-01T20:51:31.484+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1ac scrub starts
2023-11-01T20:51:34.455+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.17 scrub starts
2023-11-01T20:51:39.444+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1d9 deep-scrub starts
2023-11-01T20:51:42.473+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 5.9b scrub starts
2023-11-01T20:51:44.510+0000 7f3be0b27700 0 log_channel(cluster)
log [DBG] : 5.65 scrub starts
2023-11-01T20:51:46.491+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 6.2d scrub starts
2023-11-01T20:51:47.465+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1ac scrub starts
2023-11-01T20:51:49.443+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.17 scrub starts
2023-11-01T20:51:51.439+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1d9 scrub starts
2023-11-01T20:51:53.388+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 5.9b scrub starts
2023-11-01T20:52:00.345+0000 7f3be0b27700 0 log_channel(cluster)
log [DBG] : 5.65 scrub starts
2023-11-01T20:52:02.438+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 6.2d scrub starts
2023-11-01T20:52:03.452+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1ac scrub starts
2023-11-01T20:52:10.421+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.17 scrub starts
2023-11-01T20:52:11.436+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1d9 deep-scrub starts
2023-11-01T20:52:12.465+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 5.9b deep-scrub starts
2023-11-01T20:52:13.470+0000 7f3be0b27700 0 log_channel(cluster)
log [DBG] : 5.65 scrub starts
2023-11-01T20:52:14.468+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 6.2d deep-scrub starts
2023-11-01T20:52:17.512+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1ac scrub starts
2023-11-01T20:52:20.507+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.17 scrub starts
2023-11-01T20:52:22.428+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1d9 scrub starts
2023-11-01T20:52:23.438+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 5.9b scrub starts
2023-11-01T20:52:24.444+0000 7f3be0b27700 0 log_channel(cluster)
log [DBG] : 5.65 scrub starts
2023-11-01T20:52:28.461+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 6.2d scrub starts
2023-11-01T20:52:45.551+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1ac scrub starts
2023-11-01T20:52:46.593+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.17 scrub starts
2023-11-01T20:52:48.595+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1d9 scrub starts
2023-11-01T20:52:52.488+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 5.9b deep-scrub starts
2023-11-01T20:52:55.504+0000 7f3be0b27700 0 log_channel(cluster)
log [DBG] : 5.65 scrub starts
2023-11-01T20:52:58.519+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 6.2d scrub starts
2023-11-01T20:52:59.477+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1ac scrub starts
2023-11-01T20:53:01.505+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.17 scrub starts
2023-11-01T20:53:03.467+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1d9 scrub starts
2023-11-01T20:53:06.406+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 5.9b scrub starts
2023-11-01T20:53:10.446+0000 7f3be0b27700 0 log_channel(cluster)
log [DBG] : 5.65 scrub starts
2023-11-01T20:53:13.470+0000 7f3be1328700 0 log_channel(cluster)
log [DBG] : 6.2d scrub starts
2023-11-01T20:53:15.521+0000 7f3bdfb25700 0 log_channel(cluster)
log [DBG] : 5.1ac scrub starts
There is no scrub OK type messages.
I notice these logs were from yesterday, and currently there is
nothing being logged for these OSDs. If I was to "kick the cluster"
these scrub logs would probably start showing up again.
When this is happening the PGs mentioned seem to stay in a "scrub
queued" state.
Any light you could shine my way would be appreciated.
Thanks,
Martin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
200
days inactive
206
days old
8 comments
4 participants
participants (4)
-
Eugen Block -
John Mulligan -
Martin Conway -
martin.conway＠anu.edu.au