Hello,
for those sumbling upon a similar issue: I was able to mitigate the
issue, by setting
=== 8< ===
[osd.14]
osd_pg_max_concurrent_snap_trims = 0
=========
in ceph.conf. You don't need to restart the osd, osd crash crash +
systemd will do it for you :)
Now the osd in question does no trimming anymore and thus stays up.
Now I let the deep-scrubber run, and press thumbs it will clean up the
mess.
In case I need to clean up manually, could anyone give a hint how to
find the rbd with that snap? The logs says:
7faf8f716700 -1 log_channel(cluster) log [ERR] : trim_object Snap 29c44
not in clones
1.) What is the 7faf8f716700 at the beginning of the log? Is it a daemon
id?
2.) About the Snap "ID" 29c44: In the filesystem I see
...ceph-14/current/7.374_head/DIR_4/DIR_7/DIR_B/DIR_A/rbd\udata.59cb9c679e2a9e3.0000000000003096__29c44_A29AAB74__7
Do I read it correctly that in PG 7.374 there is with rbd prefix
59cb9c679e2a9e3 an object that ends with ..3096, which has a snap ID
29c44 ... ? What does the part A29AAB74__7 ?
I was nit able to find in docs how the directory / filename is structured.
Best Regrads,
j.
On 31.01.20 16:04, Andreas John wrote:
Hello,
in my cluster one after the other OSD dies until I recognized that it
was simply an "abort" in the daemon caused probably by
2020-01-31 15:54:42.535930 7faf8f716700 -1 log_channel(cluster) log
[ERR] : trim_object Snap 29c44 not in clones
Close to this msg I get a stracktrace:
ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
1: /usr/bin/ceph-osd() [0xb35f7d]
2: (()+0x11390) [0x7f0fec74b390]
3: (gsignal()+0x38) [0x7f0feab43428]
4: (abort()+0x16a) [0x7f0feab4502a]
5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f0feb48684d]
6: (()+0x8d6b6) [0x7f0feb4846b6]
7: (()+0x8d701) [0x7f0feb484701]
8: (()+0x8d919) [0x7f0feb484919]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x27e) [0xc3776e]
10: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x10dd) [0x868cfd]
11: (ReplicatedPG::repop_all_committed(ReplicatedPG::RepGather*)+0x80)
[0x8690e0]
12: (Context::complete(int)+0x9) [0x6c8799]
13: (void ReplicatedBackend::sub_op_modify_reply<MOSDRepOpReply,
113>(std::tr1::shared_ptr<OpRequest>)+0x21b) [0xa5ae0b]
14:
(ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x15b)
[0xa53edb]
15: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x1cb) [0x84c78b]
16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3ef) [0x6966ff]
17: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x4e4) [0x696e14]
18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x71e)
[0xc264fe]
19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xc29950]
20: (()+0x76ba) [0x7f0fec7416ba]
21: (clone()+0x6d) [0x7f0feac1541d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
Yes, I know it's still hammer, I want to upgrade soon, but I want to
resolve that issue first. If I lose that PG, I don't worry.
So: What it the best approach? Can I use something like
ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid> ? I
assume 29c44 is my Object, but what's the clone od?
Best regards,
derjohn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Andreas John
net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach
Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832
Tel: +49 69 8570033-1 | Fax: -2 |
http://www.net-lab.net
Facebook:
https://www.facebook.com/netlabdotnet
Twitter:
https://twitter.com/netlabdotnet