"cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"
Hi I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses. I still can't get the reef version of cephadm to work properly. I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email. Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version" Traceback (most recent call last): File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9468, in <module> main() File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9456, in main r = ctx.func(ctx) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2108, in _infer_image ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2191, in infer_local_ceph_image container_info = get_container_info(ctx, daemon, daemon_name is not None) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in get_container_info matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in <listcomp> matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 217, in __getattr__ return super().__getattribute__(name) AttributeError: 'CephadmContext' object has no attribute 'fsid' I am running into other issues as well, but I think they may point back to this issue of "'CephadmContext' object has no attribute 'fsid'" Any help would be appreciated. Regards, Martin Conway IT and Digital Media Manager Research School of Physics Australian National University Canberra ACT 2601 +61 2 6125 1599 https://physics.anu.edu.au<https://physics.anu.edu.au/>
I just had another look through the issues tracker and found this bug already listed. https://tracker.ceph.com/issues/59428 I need to go back to the other issues I am having and figure out if they are related or something different. Hi I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses. I still can't get the reef version of cephadm to work properly. I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email. Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version" Traceback (most recent call last): File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9468, in <module> main() File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9456, in main r = ctx.func(ctx) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2108, in _infer_image ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2191, in infer_local_ceph_image container_info = get_container_info(ctx, daemon, daemon_name is not None) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in get_container_info matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in <listcomp> matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 217, in __getattr__ return super().__getattribute__(name) AttributeError: 'CephadmContext' object has no attribute 'fsid'
Are the issues you refer to the same as before? I don't think this version issue is the root cause, I do see it as well in my test cluster(s) but the rest works properly except for the tag issue I already reported which you can easily fix by setting the config value for the default image (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGDY...). Or are there new issues you encountered? Zitat von Martin Conway <martin.conway@anu.edu.au>:
I just had another look through the issues tracker and found this bug already listed. https://tracker.ceph.com/issues/59428
I need to go back to the other issues I am having and figure out if they are related or something different.
Hi
I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses.
I still can't get the reef version of cephadm to work properly.
I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email.
Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version"
Traceback (most recent call last): File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9468, in <module> main() File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9456, in main r = ctx.func(ctx) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2108, in _infer_image ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2191, in infer_local_ceph_image container_info = get_container_info(ctx, daemon, daemon_name is not None) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in get_container_info matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in <listcomp> matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 217, in __getattr__ return super().__getattribute__(name) AttributeError: 'CephadmContext' object has no attribute 'fsid'
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Friday, October 27, 2023 2:40:17 AM EDT Eugen Block wrote:
Are the issues you refer to the same as before? I don't think this version issue is the root cause, I do see it as well in my test cluster(s) but the rest works properly except for the tag issue I already reported which you can easily fix by setting the config value for the default image (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGD YAWPVE2YLV2ZLF3HC5SLS/#LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS). Or are there new issues you encountered?
I concur. That `cephadm version` failure is/was a known issue but should not be the cause of any other issues. On the main branch `cephadm version` no longer fails this way - rather, it reports the version of a cephadm build and no longer inspects a container image. We can look into backporting this before the next reef release. The issue related to the container image tag that Eugen filed has also been fixed on reef. Thanks for filing that. Martin you may want to retry things after the next reef release. Unfortunately, I don't know when that is planned but I think it's soonish.
Zitat von Martin Conway <martin.conway@anu.edu.au>:
I just had another look through the issues tracker and found this bug already listed. https://tracker.ceph.com/issues/59428
I need to go back to the other issues I am having and figure out if they are related or something different.
Hi
I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses.
I still can't get the reef version of cephadm to work properly.
I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email.
Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version"
Traceback (most recent call last): File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 e", line 9468, in <module>
main()
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 e", line 9456, in main
r = ctx.func(ctx)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 e", line 2108, in _infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 e", line 2191, in infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not None)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 e", line 2154, in get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 e", line 2154, in <listcomp>
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4 e", line 217, in __getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry for the delayed response. Office365 keeps quarantining 75% of the emails from this list. I had a couple of other issues, one of which seems to have come good by itself. cephadm adopt --style legacy --name osd.12 Would fail, but I tried it again yesterday without having made any other changes I can think of, and it worked fine. Sorry, I don't have a record of what the error message was. I have been trying to add SSD DB/WAL to all of my Bluestor OSD spinning disks. The commands I found to simply move the DB didn't work for me so I have been removing each OSD and recreating again using ceph-colume, then adopting again. This process has been going on for weeks, and I now have a persistent issue with scrubs not having been completed in time. I am unsure if this will sort itself out if I just leave it alone for a week or two, or if there is an underlying issue with Reef and scrubbing being slow. It definitely doesn't fix itself if left alone for a day or two. I find that backfilling and possibly scrubbing often comes to a halt for no apparent reason. If I put a server into maintenance mode or kill and restart OSDs it bursts back into life again. Not sure how to diagnose why the recovery processes have stalled. Regards, Martin
-----Original Message----- From: John Mulligan <phlogistonjohn@asynchrono.us> Sent: Saturday, October 28, 2023 12:58 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: "cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"
[You don't often get email from phlogistonjohn@asynchrono.us. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
On Friday, October 27, 2023 2:40:17 AM EDT Eugen Block wrote:
Are the issues you refer to the same as before? I don't think this version issue is the root cause, I do see it as well in my test cluster(s) but the rest works properly except for the tag issue I already reported which you can easily fix by setting the config value for the default image (https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis ts.ceph.io%2Fhyperkitty%2Flist%2Fceph- users%40ceph.io%2Fthread%2FLASBJ
CSPFGD&data=05%7C01%7Cmartin.conway%40anu.edu.au%7Cbc11333aca 854f478a0
308dbd6f4d47e%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C 63834310466
8081112%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi V2luMzIiLC
JBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iaeVqSYY8 %2B6a8mqZ1
hgGmGif5%2BAjrWgXqeuhGaRKy7Q%3D&reserved=0 YAWPVE2YLV2ZLF3HC5SLS/#LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS). Or are there new issues you encountered?
I concur. That `cephadm version` failure is/was a known issue but should not be the cause of any other issues. On the main branch `cephadm version` no longer fails this way - rather, it reports the version of a cephadm build and no longer inspects a container image. We can look into backporting this before the next reef release.
The issue related to the container image tag that Eugen filed has also been fixed on reef. Thanks for filing that.
Martin you may want to retry things after the next reef release. Unfortunately, I don't know when that is planned but I think it's soonish.
Zitat von Martin Conway <martin.conway@anu.edu.au>:
I just had another look through the issues tracker and found this bug already listed. https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftr
acker.ceph.com%2Fissues%2F59428&data=05%7C01%7Cmartin.conway%40 anu.e
du.au%7Cbc11333aca854f478a0308dbd6f4d47e%7Ce37d725cab5c46249a e5f0533
e486437%7C0%7C0%7C638343104668081112%7CUnknown%7CTWFpbGZ sb3d8eyJWIjo
iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3 000%7
C%7C%7C&sdata=kYRi9R4hSa5CPErQEct5MRUftPG%2FpFcLHu30%2FfpWUQ w%3D&res
erved=0
I need to go back to the other issues I am having and figure out if they are related or something different.
Hi
I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses.
I still can't get the reef version of cephadm to work properly.
I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email.
Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version"
Traceback (most recent call last): File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 9468, in <module>
main()
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 9456, in main
r = ctx.func(ctx)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 2108, in _infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 2191, in infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not None)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 2154, in get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 2154, in <listcomp>
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 217, in __getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Martin Conway wrote:
I find that backfilling and possibly scrubbing often comes to a halt for no apparent reason. If I put a server into maintenance mode or kill and restart OSDs it bursts back into life again.
Not sure how to diagnose why the recovery processes have stalled.
My cluster is in this stalled state now, I have saved some details below. Seems to point quite heavily to OSD.32 and OSD.33 but there is nothing of note in their logs. They were experiencing slow ops last night, and this morning have logged nothing. I am certain recovery and scrubbing will resume if I restarted those OSDs, but it would be nice to know what keep causing this. ceph -s cluster: id: 16bb4f7a-cf04-4667-aeee-94ce7f6ab672 health: HEALTH_WARN 441 pgs not deep-scrubbed in time 43 pgs not scrubbed in time services: mon: 5 daemons, quorum scustor3,scustor2,scustor1,scustor4,scustor5 (age 23h) mgr: scustor3.wplaov(active, since 2d), standbys: scustor4.giyegr, scustor1.luywbi, scustor2.ncfaec mds: 2/2 daemons up, 1 standby osd: 31 osds: 31 up (since 23h), 31 in (since 4d); 54 remapped pgs data: volumes: 1/1 healthy pools: 8 pools, 897 pgs objects: 48.52M objects, 47 TiB usage: 130 TiB used, 130 TiB / 260 TiB avail pgs: 3497437/145569867 objects misplaced (2.403%) 757 active+clean 70 active+clean+scrubbing 48 active+remapped+backfilling 14 active+clean+scrubbing+deep 6 active+recovering+remapped 2 active+recovering io: client: 24 KiB/s rd, 416 KiB/s wr, 1 op/s rd, 44 op/s wr ceph pg dump https://pastebin.com/raw/KPBie7SD ceph pg dump_stuck PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 5.1f4 active+remapped+backfilling [32,5,18] 32 [32,5,21] 32 5.1f0 active+remapped+backfilling [32,2,13] 32 [32,2,5] 32 5.1e4 active+remapped+backfilling [33,13,10] 33 [33,6,5] 33 5.1c5 active+remapped+backfilling [33,2,10] 33 [33,10,16] 33 5.1bc active+remapped+backfilling [18,33,11] 18 [33,11,6] 33 5.193 active+remapped+backfilling [33,14,3] 33 [33,13,5] 33 5.180 active+remapped+backfilling [32,13,1] 32 [32,1,10] 32 5.171 active+remapped+backfilling [33,1,18] 33 [33,1,20] 33 5.16b active+recovering+remapped [32,9,13] 32 [32,9,2] 32 5.16a active+remapped+backfilling [13,4,6] 13 [33,4,16] 33 5.169 active+remapped+backfilling [14,33,10] 14 [33,10,13] 33 5.162 active+remapped+backfilling [32,6,13] 32 [32,6,22] 32 5.130 active+remapped+backfilling [13,32,3] 13 [32,3,1] 32 5.1cd active+remapped+backfilling [33,5,18] 33 [33,5,13] 33 6.3b active+recovering [32,1,3] 32 [32,1,3] 32 6.42 active+remapped+backfilling [18,33,9] 18 [33,9,13] 33 5.4e active+remapped+backfilling [32,11,10] 32 [32,10,22] 32 5.167 active+remapped+backfilling [33,18,6] 33 [33,6,2] 33 6.20 active+remapped+backfilling [14,32,10] 14 [32,13,1] 32 5.52 active+remapped+backfilling [14,1,32] 14 [32,1,13] 32 5.57 active+recovering+remapped [32,9,13] 32 [32,9,22] 32 5.49 active+remapped+backfilling [18,32,9] 18 [32,9,13] 32 5.1f7 active+recovering+remapped [9,32,13] 9 [32,20,1] 32 5.100 active+remapped+backfilling [33,9,13] 33 [33,9,20] 33 5.58 active+remapped+backfilling [32,6,4] 32 [32,6,20] 32 5.16 active+recovering+remapped [32,18,3] 32 [32,3,9] 32 5.60 active+recovering+remapped [31,13,4] 31 [32,4,9] 32 5.fc active+recovering [32,9,10] 32 [32,9,10] 32 5.c8 active+remapped+backfilling [32,14,3] 32 [32,3,13] 32 5.ad active+remapped+backfilling [32,3,16] 32 [32,3,22] 32 5.6e active+remapped+backfilling [32,5,18] 32 [32,5,11] 32 6.6d active+remapped+backfilling [32,6,5] 32 [32,5,21] 32 5.cf active+remapped+backfilling [33,9,18] 33 [33,11,16] 33 5.7e active+remapped+backfilling [32,9,18] 32 [32,4,16] 32 6.37 active+remapped+backfilling [32,4,3] 32 [32,4,21] 32 5.1aa active+remapped+backfilling [13,33,10] 13 [33,10,1] 33 5.165 active+remapped+backfilling [33,5,16] 33 [33,5,21] 33 5.76 active+remapped+backfilling [13,1,32] 13 [32,1,22] 32 5.102 active+remapped+backfilling [33,5,6] 33 [33,5,21] 33 5.2d active+remapped+backfilling [32,18,4] 32 [32,4,5] 32 6.24 active+remapped+backfilling [33,18,2] 33 [33,9,3] 33 5.f6 active+remapped+backfilling [32,1,14] 32 [32,1,22] 32 5.1c active+remapped+backfilling [33,18,3] 33 [33,3,22] 33 5.d9 active+remapped+backfilling [33,18,11] 33 [33,11,9] 33 5.184 active+remapped+backfilling [32,14,2] 32 [32,20,5] 32 5.e6 active+remapped+backfilling [18,33,16] 18 [33,16,13] 33 5.18f active+recovering+remapped [18,32,9] 18 [32,9,13] 32 5.e9 active+remapped+backfilling [32,13,9] 32 [32,13,2] 32 5.55 active+remapped+backfilling [32,6,14] 32 [32,6,3] 32 5.eb active+remapped+backfilling [18,33,11] 18 [32,20,6] 32 6.13 active+remapped+backfilling [14,10,1] 14 [32,20,13] 32 5.107 active+remapped+backfilling [14,3,31] 14 [32,3,1] 32 5.109 active+remapped+backfilling [32,4,14] 32 [32,4,13] 32 5.117 active+remapped+backfilling [33,16,3] 33 [33,16,20] 33 6.30 active+remapped+backfilling [32,4,1] 32 [32,1,21] 32 5.126 active+remapped+backfilling [33,9,4] 33 [33,9,21] 33 ok
Hi, first of all, I'd still recommend to use the orchestrator to deploy OSDs. Building OSDs manually and then adopt them is redundant. Or do you have issues with the drivegroups? I don't have *the* solution but you could try to disable the mclock scheduler [1] which is the default since Quincy. Maybe that will speed up things? There have been reports in the list about some unwanted or at least unexpected behavior. As for the "not (deep-)scrubbed in time" messages, there seems to be progress (in your ceph status), but depending on the drive utilization you could increase the number of scrubs per OSD (osd_max_scrubs). Regards, Eugen [1] https://www.clyso.com/blog/ceph-how-do-disable-mclock-scheduler/ Zitat von Martin Conway <martin.conway@anu.edu.au>:
Sorry for the delayed response. Office365 keeps quarantining 75% of the emails from this list.
I had a couple of other issues, one of which seems to have come good by itself.
cephadm adopt --style legacy --name osd.12
Would fail, but I tried it again yesterday without having made any other changes I can think of, and it worked fine. Sorry, I don't have a record of what the error message was.
I have been trying to add SSD DB/WAL to all of my Bluestor OSD spinning disks. The commands I found to simply move the DB didn't work for me so I have been removing each OSD and recreating again using ceph-colume, then adopting again. This process has been going on for weeks, and I now have a persistent issue with scrubs not having been completed in time. I am unsure if this will sort itself out if I just leave it alone for a week or two, or if there is an underlying issue with Reef and scrubbing being slow. It definitely doesn't fix itself if left alone for a day or two.
I find that backfilling and possibly scrubbing often comes to a halt for no apparent reason. If I put a server into maintenance mode or kill and restart OSDs it bursts back into life again.
Not sure how to diagnose why the recovery processes have stalled.
Regards, Martin
-----Original Message----- From: John Mulligan <phlogistonjohn@asynchrono.us> Sent: Saturday, October 28, 2023 12:58 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: "cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"
[You don't often get email from phlogistonjohn@asynchrono.us. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
On Friday, October 27, 2023 2:40:17 AM EDT Eugen Block wrote:
Are the issues you refer to the same as before? I don't think this version issue is the root cause, I do see it as well in my test cluster(s) but the rest works properly except for the tag issue I already reported which you can easily fix by setting the config value for the default image (https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis ts.ceph.io%2Fhyperkitty%2Flist%2Fceph- users%40ceph.io%2Fthread%2FLASBJ
CSPFGD&data=05%7C01%7Cmartin.conway%40anu.edu.au%7Cbc11333aca 854f478a0
308dbd6f4d47e%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C 63834310466
8081112%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi V2luMzIiLC
JBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iaeVqSYY8 %2B6a8mqZ1
hgGmGif5%2BAjrWgXqeuhGaRKy7Q%3D&reserved=0 YAWPVE2YLV2ZLF3HC5SLS/#LASBJCSPFGDYAWPVE2YLV2ZLF3HC5SLS). Or are there new issues you encountered?
I concur. That `cephadm version` failure is/was a known issue but should not be the cause of any other issues. On the main branch `cephadm version` no longer fails this way - rather, it reports the version of a cephadm build and no longer inspects a container image. We can look into backporting this before the next reef release.
The issue related to the container image tag that Eugen filed has also been fixed on reef. Thanks for filing that.
Martin you may want to retry things after the next reef release. Unfortunately, I don't know when that is planned but I think it's soonish.
Zitat von Martin Conway <martin.conway@anu.edu.au>:
I just had another look through the issues tracker and found this bug already listed. https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftr
acker.ceph.com%2Fissues%2F59428&data=05%7C01%7Cmartin.conway%40 anu.e
du.au%7Cbc11333aca854f478a0308dbd6f4d47e%7Ce37d725cab5c46249a e5f0533
e486437%7C0%7C0%7C638343104668081112%7CUnknown%7CTWFpbGZ sb3d8eyJWIjo
iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3 000%7
C%7C%7C&sdata=kYRi9R4hSa5CPErQEct5MRUftPG%2FpFcLHu30%2FfpWUQ w%3D&res
erved=0
I need to go back to the other issues I am having and figure out if they are related or something different.
Hi
I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses.
I still can't get the reef version of cephadm to work properly.
I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email.
Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version"
Traceback (most recent call last): File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 9468, in <module>
main()
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 9456, in main
r = ctx.func(ctx)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 2108, in _infer_image
ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 2191, in infer_local_ceph_image
container_info = get_container_info(ctx, daemon, daemon_name is not None)
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 2154, in get_container_info
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 2154, in <listcomp>
matching_daemons = [d for d in daemons if daemon_name_or_type(d)
== daemon_filter and d['fsid'] == ctx.fsid]
File
"./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc1 05c
afd9b4 e", line 217, in __getattr__
return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
first of all, I'd still recommend to use the orchestrator to deploy OSDs. Building OSDs manually and then adopt them is redundant. Or do you have issues with the drivegroups?
I am having to do it this way because I couldn't find any doco on how to specify a separate DB/WAL device when deploying OSDs using the orchestrator. If there is such a command I agree it would be a better choice. Ideally I would use the commands to simply move the DB of my existing orchestrator deployed ODSs to the SSD, but when I tried that command it broke my OSD and I had to delete it and leave he cluster in a degraded state until it had recovered. I find it very stressful when I get out of my depth with problems like that, so I gave up on that idea and am doing the remove, redploy, adopt method, which is working, but VERY slow.
I don't have *the* solution but you could try to disable the mclock scheduler [1] which is the default since Quincy. Maybe that will speed up things? There have been reports in the list about some unwanted or at least unexpected behavior.
I did try this to try and speed up my rebalances, but it didn't seem to make much difference. I haven't tried it to see what difference it makes to scrubbing.
As for the "not (deep-)scrubbed in time" messages, there seems to be progress (in your ceph status), but depending on the drive utilization you could increase the number of scrubs per OSD (osd_max_scrubs).
There are lot of scrubs running, this morning after my rebalance finally completed it has 22 scrubbing, 6 deep scrubbing (across 28 OSDs). This has fallen from the number it was running yesterday when the rebalance was still happening (38/9). I believe if I kick the cluster by taking a host into maintenance and back the numbers will jump up again. The trouble is I don't know how to tell if a scrub is actually achieving something, stuck or restarting over and over. My current ceph pg dump is: https://pastebin.com/AQhNKSBN and if I run it again a few minutes later: https://pastebin.com/yfREzJ4s I see evidence of scrubs not working because some of my OSD logs look like this: 2023-11-01T20:51:08.668+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:11.658+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:19.565+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:20.516+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:51:22.463+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:51:24.488+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:51:29.474+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:31.484+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:34.455+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:39.444+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 deep-scrub starts 2023-11-01T20:51:42.473+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:51:44.510+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:51:46.491+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:47.465+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:49.443+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:51.439+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:51:53.388+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:52:00.345+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:02.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:03.452+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:10.421+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:11.436+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 deep-scrub starts 2023-11-01T20:52:12.465+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b deep-scrub starts 2023-11-01T20:52:13.470+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:14.468+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d deep-scrub starts 2023-11-01T20:52:17.512+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:20.507+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:22.428+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:52:23.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:52:24.444+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:28.461+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:45.551+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:46.593+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:48.595+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:52:52.488+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b deep-scrub starts 2023-11-01T20:52:55.504+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:58.519+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:59.477+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:53:01.505+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:53:03.467+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:53:06.406+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:53:10.446+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:53:13.470+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:53:15.521+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts There is no scrub OK type messages. I notice these logs were from yesterday, and currently there is nothing being logged for these OSDs. If I was to "kick the cluster" these scrub logs would probably start showing up again. When this is happening the PGs mentioned seem to stay in a "scrub queued" state. Any light you could shine my way would be appreciated. Thanks, Martin
There are a couple of examples in the docs [2], so in your case it probably would be something rather simple like: service_type: osd service_id: osd_spec_default placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 You can apply that config to specific hosts or all of them, it really depends on your actual setup. You can also dry-run the config before applying it with the --dry-run flag: ceph orch apply -i my-osd-specs.yaml --dry-run I'd recommend to create a test cluster if possible to have some options to practice and get familiar with all that stuff.
Ideally I would use the commands to simply move the DB of my existing orchestrator deployed ODSs to the SSD, but when I tried that command it broke my OSD and I had to delete it and leave he cluster in a degraded state until it had recovered.
Do you still have the commands and the output somewhere what exactly went wrong? I haven't migrated DBs in quite some time, especially not in a newer version like Reef. I assume you tried it with bluefs-bdev-migrate? I remember having seen these repeating scrub starts messages in this list, but I can't seem to find the right thread. I can't recall if there was a solution to that... When was the last time you failed the mgr service? That still does help sometimes... [2] https://docs.ceph.com/en/reef/cephadm/services/osd/#examples Zitat von Martin Conway <martin.conway@anu.edu.au>:
first of all, I'd still recommend to use the orchestrator to deploy OSDs. Building OSDs manually and then adopt them is redundant. Or do you have issues with the drivegroups?
I am having to do it this way because I couldn't find any doco on how to specify a separate DB/WAL device when deploying OSDs using the orchestrator. If there is such a command I agree it would be a better choice.
Ideally I would use the commands to simply move the DB of my existing orchestrator deployed ODSs to the SSD, but when I tried that command it broke my OSD and I had to delete it and leave he cluster in a degraded state until it had recovered. I find it very stressful when I get out of my depth with problems like that, so I gave up on that idea and am doing the remove, redploy, adopt method, which is working, but VERY slow.
I don't have *the* solution but you could try to disable the mclock scheduler [1] which is the default since Quincy. Maybe that will speed up things? There have been reports in the list about some unwanted or at least unexpected behavior.
I did try this to try and speed up my rebalances, but it didn't seem to make much difference. I haven't tried it to see what difference it makes to scrubbing.
As for the "not (deep-)scrubbed in time" messages, there seems to be progress (in your ceph status), but depending on the drive utilization you could increase the number of scrubs per OSD (osd_max_scrubs).
There are lot of scrubs running, this morning after my rebalance finally completed it has 22 scrubbing, 6 deep scrubbing (across 28 OSDs). This has fallen from the number it was running yesterday when the rebalance was still happening (38/9).
I believe if I kick the cluster by taking a host into maintenance and back the numbers will jump up again. The trouble is I don't know how to tell if a scrub is actually achieving something, stuck or restarting over and over.
My current ceph pg dump is: https://pastebin.com/AQhNKSBN
and if I run it again a few minutes later: https://pastebin.com/yfREzJ4s
I see evidence of scrubs not working because some of my OSD logs look like this: 2023-11-01T20:51:08.668+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:11.658+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:19.565+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:20.516+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:51:22.463+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:51:24.488+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:51:29.474+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:31.484+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:34.455+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:39.444+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 deep-scrub starts 2023-11-01T20:51:42.473+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:51:44.510+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:51:46.491+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:47.465+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:49.443+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:51.439+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:51:53.388+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:52:00.345+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:02.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:03.452+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:10.421+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:11.436+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 deep-scrub starts 2023-11-01T20:52:12.465+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b deep-scrub starts 2023-11-01T20:52:13.470+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:14.468+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d deep-scrub starts 2023-11-01T20:52:17.512+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:20.507+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:22.428+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:52:23.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:52:24.444+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:28.461+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:45.551+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:46.593+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:48.595+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:52:52.488+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b deep-scrub starts 2023-11-01T20:52:55.504+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:58.519+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:59.477+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:53:01.505+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:53:03.467+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:53:06.406+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:53:10.446+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:53:13.470+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:53:15.521+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts
There is no scrub OK type messages.
I notice these logs were from yesterday, and currently there is nothing being logged for these OSDs. If I was to "kick the cluster" these scrub logs would probably start showing up again.
When this is happening the PGs mentioned seem to stay in a "scrub queued" state.
Any light you could shine my way would be appreciated.
Thanks, Martin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
John Mulligan
-
Martin Conway
-
martin.conway＠anu.edu.au