Re: How can I fix "object unfound" error?
Maybe.... I've marked the object as "lost" and removed the failed OSD.
The cluster now is healthy, but I'd like to understand if it's likely to bother me again in the future. Yeah, I don't know.
Within the last month there are 4 separate instances of people mentioning "unfound" object in their cluster. I'm deferring as long as possible any OSD drive upgrades. I ran into the problem when "draining" an OSD. "draining" means remove OSD from crush map, wait for all PG to be stored elsewhere, then replace drive with larger one. Under those circumstances there should be no PG unfound. BTW, are you using cache tiering ? The bug report mentions this, but some people did not have this enabled. Chad.
In data mercoledì 4 marzo 2020 18:14:31 CET, Chad William Seys ha scritto:
Maybe.... I've marked the object as "lost" and removed the failed OSD.
The cluster now is healthy, but I'd like to understand if it's likely to bother me again in the future.
Yeah, I don't know.
Within the last month there are 4 separate instances of people mentioning "unfound" object in their cluster.
I'm deferring as long as possible any OSD drive upgrades. I ran into the problem when "draining" an OSD.
"draining" means remove OSD from crush map, wait for all PG to be stored elsewhere, then replace drive with larger one. Under those circumstances there should be no PG unfound.
BTW, are you using cache tiering ? The bug report mentions this, but some people did not have this enabled.
Chad.
No, I don't have cache tiering enabled. I also found strange that the PG was marked unfound: the cluster was perfectly healthy before the kernel panic and a single OSD failure shouldn't create mush hassle. *Simone Lazzaris* *Qcom S.p.A. a Socio Unico* Via Roggia Vignola, 9 | 24047 Treviglio (BG)T +39 0363 1970352 | M +39 3938111237 simone.lazzaris@qcom.it[1] | www.qcom.it[2] * LinkedIn[3]* | *Facebook*[4] [5] -------- [1] mailto:simone.lazzaris@qcom.it [2] https://www.qcom.it [3] https://www.linkedin.com/company/qcom-spa [4] http://www.facebook.com/qcomspa [5] https://www.qcom.it/includes/NUOVAemail-banner.gif
No, I don't have cache tiering enabled. I also found strange that the PG was marked unfound: the cluster was perfectly healthy before the kernel panic and a single OSD failure shouldn't create mush hassle.
Yes, it is a bug unless using a singly replicated pool! C.
Simone; What is your failure domain? If you don't know your failure domain can you provide the CRUSH ruleset for the pool that experienced the "object unfound" error? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Simone Lazzaris [mailto:simone.lazzaris@qcom.it] Sent: Thursday, March 05, 2020 6:11 AM To: ceph-users; Chad William Seys Subject: [ceph-users] Re: How can I fix "object unfound" error? In data mercoledì 4 marzo 2020 18:14:31 CET, Chad William Seys ha scritto:
Maybe.... I've marked the object as "lost" and removed the failed OSD.
The cluster now is healthy, but I'd like to understand if it's likely to bother me again in the future.
Yeah, I don't know.
Within the last month there are 4 separate instances of people mentioning "unfound" object in their cluster.
I'm deferring as long as possible any OSD drive upgrades. I ran into the problem when "draining" an OSD.
"draining" means remove OSD from crush map, wait for all PG to be stored elsewhere, then replace drive with larger one. Under those circumstances there should be no PG unfound.
BTW, are you using cache tiering ? The bug report mentions this, but some people did not have this enabled.
Chad.
No, I don't have cache tiering enabled. I also found strange that the PG was marked unfound: the cluster was perfectly healthy before the kernel panic and a single OSD failure shouldn't create mush hassle. *Simone Lazzaris* *Qcom S.p.A. a Socio Unico* Via Roggia Vignola, 9 | 24047 Treviglio (BG)T +39 0363 1970352 | M +39 3938111237 simone.lazzaris@qcom.it[1] | www.qcom.it[2] * LinkedIn[3]* | *Facebook*[4] [5] -------- [1] mailto:simone.lazzaris@qcom.it [2] https://www.qcom.it [3] https://www.linkedin.com/company/qcom-spa [4] http://www.facebook.com/qcomspa [5] https://www.qcom.it/includes/NUOVAemail-banner.gif _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
In data giovedì 5 marzo 2020 23:12:49 CET, DHilsbos@performair.com ha scritto:
Simone;
What is your failure domain?
If you don't know your failure domain can you provide the CRUSH ruleset for the pool that experienced the "object unfound" error?
Thank you,
As far as I can tell (not *exactly* an expert here), my failre domain is set to host, with a simple erasure code profile: root@s1:~# ceph osd erasure-code-profile ls default root@s1:~# ceph osd erasure-code-profile get default k=2 m=1 plugin=jerasure technique=reed_sol_van root@s1:~# ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "erasure-code", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] The pool was using the erasure code crush map, not the replicated one. *Simone Lazzaris* *Qcom S.p.A. a Socio Unico* Via Roggia Vignola, 9 | 24047 Treviglio (BG)T +39 0363 47905 | D +39 0363 1970352 simone.lazzaris@qcom.it[1] | www.qcom.it[2] * LinkedIn[3]* | *Facebook[4]* -------- [1] mailto:simone.lazzaris@qcom.it [2] https://www.qcom.it [3] https://www.linkedin.com/company/qcom-spa [4] http://www.facebook.com/qcomspa
participants (3)
-
Chad William Seys
-
DHilsbos＠performair.com
-
Simone Lazzaris