31 Jan
2020
31 Jan
'20
7:04 a.m.
Hello,
in my cluster one after the other OSD dies until I recognized that it
was simply an "abort" in the daemon caused probably by
2020-01-31 15:54:42.535930 7faf8f716700 -1 log_channel(cluster) log
[ERR] : trim_object Snap 29c44 not in clones
Close to this msg I get a stracktrace:
ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
1: /usr/bin/ceph-osd() [0xb35f7d]
2: (()+0x11390) [0x7f0fec74b390]
3: (gsignal()+0x38) [0x7f0feab43428]
4: (abort()+0x16a) [0x7f0feab4502a]
5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f0feb48684d]
6: (()+0x8d6b6) [0x7f0feb4846b6]
7: (()+0x8d701) [0x7f0feb484701]
8: (()+0x8d919) [0x7f0feb484919]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x27e) [0xc3776e]
10: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x10dd) [0x868cfd]
11: (ReplicatedPG::repop_all_committed(ReplicatedPG::RepGather*)+0x80)
[0x8690e0]
12: (Context::complete(int)+0x9) [0x6c8799]
13: (void ReplicatedBackend::sub_op_modify_reply<MOSDRepOpReply,
113>(std::tr1::shared_ptr<OpRequest>)+0x21b) [0xa5ae0b]
14:
(ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x15b)
[0xa53edb]
15: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x1cb) [0x84c78b]
16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3ef) [0x6966ff]
17: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x4e4) [0x696e14]
18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x71e)
[0xc264fe]
19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xc29950]
20: (()+0x76ba) [0x7f0fec7416ba]
21: (clone()+0x6d) [0x7f0feac1541d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
Yes, I know it's still hammer, I want to upgrade soon, but I want to
resolve that issue first. If I lose that PG, I don't worry.
So: What it the best approach? Can I use something like
ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid> ? I
assume 29c44 is my Object, but what's the clone od?
Best regards,
derjohn
1 Feb
1 Feb
1:20 a.m.
Hello,
for those sumbling upon a similar issue: I was able to mitigate the
issue, by setting
=== 8< ===
[osd.14]
osd_pg_max_concurrent_snap_trims = 0
=========
in ceph.conf. You don't need to restart the osd, osd crash crash +
systemd will do it for you :)
Now the osd in question does no trimming anymore and thus stays up.
Now I let the deep-scrubber run, and press thumbs it will clean up the
mess.
In case I need to clean up manually, could anyone give a hint how to
find the rbd with that snap? The logs says:
7faf8f716700 -1 log_channel(cluster) log [ERR] : trim_object Snap 29c44
not in clones
1.) What is the 7faf8f716700 at the beginning of the log? Is it a daemon
id?
2.) About the Snap "ID" 29c44: In the filesystem I see
...ceph-14/current/7.374_head/DIR_4/DIR_7/DIR_B/DIR_A/rbd\udata.59cb9c679e2a9e3.0000000000003096__29c44_A29AAB74__7
Do I read it correctly that in PG 7.374 there is with rbd prefix
59cb9c679e2a9e3 an object that ends with ..3096, which has a snap ID
29c44 ... ? What does the part A29AAB74__7 ?
I was nit able to find in docs how the directory / filename is structured.
Best Regrads,
j.
On 31.01.20 16:04, Andreas John wrote:
Hello,
in my cluster one after the other OSD dies until I recognized that it
was simply an "abort" in the daemon caused probably by
2020-01-31 15:54:42.535930 7faf8f716700 -1 log_channel(cluster) log
[ERR] : trim_object Snap 29c44 not in clones
Close to this msg I get a stracktrace:
ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
1: /usr/bin/ceph-osd() [0xb35f7d]
2: (()+0x11390) [0x7f0fec74b390]
3: (gsignal()+0x38) [0x7f0feab43428]
4: (abort()+0x16a) [0x7f0feab4502a]
5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f0feb48684d]
6: (()+0x8d6b6) [0x7f0feb4846b6]
7: (()+0x8d701) [0x7f0feb484701]
8: (()+0x8d919) [0x7f0feb484919]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x27e) [0xc3776e]
10: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x10dd) [0x868cfd]
11: (ReplicatedPG::repop_all_committed(ReplicatedPG::RepGather*)+0x80)
[0x8690e0]
12: (Context::complete(int)+0x9) [0x6c8799]
13: (void ReplicatedBackend::sub_op_modify_reply<MOSDRepOpReply,
113>(std::tr1::shared_ptr<OpRequest>)+0x21b) [0xa5ae0b]
14:
(ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x15b)
[0xa53edb]
15: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x1cb) [0x84c78b]
16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3ef) [0x6966ff]
17: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x4e4) [0x696e14]
18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x71e)
[0xc264fe]
19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xc29950]
20: (()+0x76ba) [0x7f0fec7416ba]
21: (clone()+0x6d) [0x7f0feac1541d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
Yes, I know it's still hammer, I want to upgrade soon, but I want to
resolve that issue first. If I lose that PG, I don't worry.
So: What it the best approach? Can I use something like
ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid> ? I
assume 29c44 is my Object, but what's the clone od?
Best regards,
derjohn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Andreas John
net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach
Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832
Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net
Facebook: https://www.facebook.com/netlabdotnet
Twitter: https://twitter.com/netlabdotnet
1:38 a.m.
Update: When repairing the PG I get a different error:
osd.14 80.69.45.76:6813/4059849 27 : cluster [INF] 7.374 repair starts
osd.14 80.69.45.76:6813/4059849 28 : cluster [ERR] 7.374 recorded data
digest 0xebbbfb83 != on disk 0x43d61c5d on
7/a29aab74/rbd_data.59cb9c679e2a9e3.0000000000003096/29c44
osd.14 80.69.45.76:6813/4059849 29 : cluster [ERR] repair 7.374
7/a29aab74/rbd_data.59cb9c679e2a9e3.0000000000003096/29c44 is an
unexpected clone
osd.14 80.69.45.76:6813/4059849 30 : cluster [ERR] 7.374 repair stat
mismatch, got 2110/2111 objects, 131/132 clones, 2110/2111 dirty, 0/0
omap, 0/0 hit_set_archive, 0/0 whiteouts, 8304141312/8304264192
bytes,0/0 hit_set_archive bytes.
osd.14 80.69.45.76:6813/4059849 31 : cluster [ERR] 7.374 repair 3
errors, 1 fixed
osd.14 80.69.45.76:6813/4059849 32 : cluster [INF] 7.374 deep-scrub starts
osd.14 80.69.45.76:6813/4059849 33 : cluster [ERR] deep-scrub 7.374
7/a29aab74/rbd_data.59cb9c679e2a9e3.0000000000003096/29c44 is an
unexpected clone
osd.14 80.69.45.76:6813/4059849 34 : cluster [ERR] 7.374 deep-scrub 1 errors
Sorry for being so noisy in list, but maybe someone can now recognize
what to do and give me a hint.
rgds.,
j
#On 01.02.20 10:20, Andreas John wrote:
Hello,
for those sumbling upon a similar issue: I was able to mitigate the
issue, by setting
=== 8< ===
[osd.14]
osd_pg_max_concurrent_snap_trims = 0
=========
in ceph.conf. You don't need to restart the osd, osd crash crash +
systemd will do it for you :)
Now the osd in question does no trimming anymore and thus stays up.
Now I let the deep-scrubber run, and press thumbs it will clean up the
mess.
In case I need to clean up manually, could anyone give a hint how to
find the rbd with that snap? The logs says:
7faf8f716700 -1 log_channel(cluster) log [ERR] : trim_object Snap 29c44
not in clones
1.) What is the 7faf8f716700 at the beginning of the log? Is it a daemon
id?
2.) About the Snap "ID" 29c44: In the filesystem I see
...ceph-14/current/7.374_head/DIR_4/DIR_7/DIR_B/DIR_A/rbd\udata.59cb9c679e2a9e3.0000000000003096__29c44_A29AAB74__7
Do I read it correctly that in PG 7.374 there is with rbd prefix
59cb9c679e2a9e3 an object that ends with ..3096, which has a snap ID
29c44 ... ? What does the part A29AAB74__7 ?
I was nit able to find in docs how the directory / filename is structured.
Best Regrads,
j.
On 31.01.20 16:04, Andreas John wrote:
> Hello,
>
> in my cluster one after the other OSD dies until I recognized that it
> was simply an "abort" in the daemon caused probably by
>
> 2020-01-31 15:54:42.535930 7faf8f716700 -1 log_channel(cluster) log
> [ERR] : trim_object Snap 29c44 not in clones
>
>
> Close to this msg I get a stracktrace:
>
>
> ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
> 1: /usr/bin/ceph-osd() [0xb35f7d]
> 2: (()+0x11390) [0x7f0fec74b390]
> 3: (gsignal()+0x38) [0x7f0feab43428]
> 4: (abort()+0x16a) [0x7f0feab4502a]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f0feb48684d]
> 6: (()+0x8d6b6) [0x7f0feb4846b6]
> 7: (()+0x8d701) [0x7f0feb484701]
> 8: (()+0x8d919) [0x7f0feb484919]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x27e) [0xc3776e]
> 10: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x10dd) [0x868cfd]
> 11: (ReplicatedPG::repop_all_committed(ReplicatedPG::RepGather*)+0x80)
> [0x8690e0]
> 12: (Context::complete(int)+0x9) [0x6c8799]
> 13: (void ReplicatedBackend::sub_op_modify_reply<MOSDRepOpReply,
> 113>(std::tr1::shared_ptr<OpRequest>)+0x21b) [0xa5ae0b]
> 14:
> (ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x15b)
> [0xa53edb]
> 15: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&,
> ThreadPool::TPHandle&)+0x1cb) [0x84c78b]
> 16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
> std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3ef) [0x6966ff]
> 17: (OSD::ShardedOpWQ::_process(unsigned int,
> ceph::heartbeat_handle_d*)+0x4e4) [0x696e14]
> 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x71e)
> [0xc264fe]
> 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xc29950]
> 20: (()+0x76ba) [0x7f0fec7416ba]
> 21: (clone()+0x6d) [0x7f0feac1541d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
>
> Yes, I know it's still hammer, I want to upgrade soon, but I want to
> resolve that issue first. If I lose that PG, I don't worry.
>
> So: What it the best approach? Can I use something like
> ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid> ? I
> assume 29c44 is my Object, but what's the clone od?
>
>
> Best regards,
>
> derjohn
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Andreas John
net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach
Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832
Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net
Facebook: https://www.facebook.com/netlabdotnet
Twitter: https://twitter.com/netlabdotnet
4:44 a.m.
Helllo,
answering to myself in case some else sutmbles upon this thread in the
future. I was able to remove the unexpected snap, here is the recipe:
How to remove the unexpected snapshots:
1.) Stop the OSD
ceph-osd -i 14 --flush-journal
... flushed journal /var/lib/ceph/osd/ceph-14/journal for object store
/var/lib/ceph/osd/ceph-14
2.) List the Object in question
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-14
--journal-path
/dev/disk/by-partuuid/212e9db1-943b-45f9-9d83-cffaeb777db7 --op list
rbd_data.59cb9c679e2a9e3.0000000000003096
[wait ... it might take minutes]
["7.374",{"oid":"rbd_data.59cb9c679e2a9e3.0000000000003096","key":"","snapid":171076,"hash":2728045428,"max":0,"pool":7,"namespace":""}]
["7.374",{"oid":"rbd_data.59cb9c679e2a9e3.0000000000003096","key":"","snapid":171797,"hash":2728045428,"max":0,"pool":7,"namespace":""}]
["7.374",{"oid":"rbd_data.59cb9c679e2a9e3.0000000000003096","key":"","snapid":-2,"hash":2728045428,"max":0,"pool":7,"namespace":""}]
3.) Remove the snap from the object
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-14
--journal-path
/dev/disk/by-partuuid/212e9db1-943b-45f9-9d83-cffaeb777db7
["7.374",{"oid":"rbd_data.59cb9c679e2a9e3.0000000000003096","key":"","snapid":171076,"hash":2728045428,"max":0,"pool":7,"namespace":""}]
remove
[wait ... it might take minutes]
remove 7/a29aab74/rbd_data.59cb9c679e2a9e3.0000000000003096/29c44
4.) Start the OSD Again
5.) Do this for all OSD on which the snap it exists. If it still exists
on one of the other OSDs, it will be synced before repair starts and
thus cause harm again.
6.) ceph pg repair 7.374
Happy again and in need of sleep,
derjohn
1544
days inactive
1545
days old
3 comments
1 participants
participants (1)
-
Andreas John