osd abort, _txc_add_transaction error
Heya, We're occasionally seeing OSD aborts with something like this in the logs (this is on 17.2.7) bluestore(/var/lib/ceph/osd/ceph-70) operator() #2:61bc4388:::3c5b76bf-81e1-4a1a-bd94-7fee95fa35e2.25380786.89__shadow_.z1wu_6Nt7uzGSnV-cca7mQxgNVyKVUm_22:head# 0x6454c6b7ab00 exists in onode_map bluestore(/var/lib/ceph/osd/ceph-70) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0) _dump_transaction transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "2.186_head", "oid": "#2:61800000::::head#" }, { "op_num": 1, "op_name": "rmcoll", "collection": "2.186_head" } ] } I see tracker issue #49900 which seems to have similar symptoms but is closed. The tracker has two Github PRs listed, #40237 and #40323 PR#40323 is merged, saying this might fix issue 49818, i.e. a different issue. PR#40237 however has been closed due to it being stale, and I wonder if tracker #49900 might have been closed erroneously? Could this be the case? Or might this be a regression? Thanks, peter.
Hi Peter, before your post I've been unaware of anything like that in 17.2.7. Generally this error means that someone is trying to delete non-empty PG. Such an attempt could be performed from other Ceph components (which we haven't seen for a while) or by a client accessing RADOS API directly. Curious if you somehow use this API? Or may be any ideas what triggers these PG removals? Data rebalancing or pool removal or something? Thanks, Igor On 11/11/2024 11:14 PM, Peter Sabaini wrote:
Heya,
We're occasionally seeing OSD aborts with something like this in the logs (this is on 17.2.7)
bluestore(/var/lib/ceph/osd/ceph-70) operator() #2:61bc4388:::3c5b76bf-81e1-4a1a-bd94-7fee95fa35e2.25380786.89__shadow_.z1wu_6Nt7uzGSnV-cca7mQxgNVyKVUm_22:head# 0x6454c6b7ab00 exists in onode_map bluestore(/var/lib/ceph/osd/ceph-70) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0) _dump_transaction transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "2.186_head", "oid": "#2:61800000::::head#" }, { "op_num": 1, "op_name": "rmcoll", "collection": "2.186_head" } ] }
I see tracker issue #49900 which seems to have similar symptoms but is closed.
The tracker has two Github PRs listed, #40237 and #40323
PR#40323 is merged, saying this might fix issue 49818, i.e. a different issue.
PR#40237 however has been closed due to it being stale, and I wonder if tracker #49900 might have been closed erroneously?
Could this be the case?
Or might this be a regression?
Thanks, peter. _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
participants (2)
-
Igor Fedotov
-
Peter Sabaini