BlueStore _txc_add_transaction errors (possibly related to bug #38724)
Hi everyone, it seems there have been several reports in the past related to BlueStore OSDs crashing from unhandled errors in _txc_add_transaction: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-April/034444.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032172.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-December/031960.htm... http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-December/031964.htm... Bug #38724 tracks this, has been fixed in master with https://github.com/ceph/ceph/pull/27929, and is pending backports (and, I dare say, is *probably* misclassified as being only minor, as this does cause potential data loss as soon as it affects enough OSDs simultaneously): https://tracker.ceph.com/issues/38724 We just ran into a similar issue with a couple of BlueStore OSDs that we recently added to a Luminous (12.2.12) cluster that was upgraded from Jewel, and hence, still largely runs on FileStore. I say similar because evidently other people reporting this problem have been running into ENOENT (No such file or directory) or ENOTEMPTY (Directory not empty); for us it's interestingly E2BIG (Argument list too long): https://tracker.ceph.com/issues/38724#note-26 So I'm wondering if someone could shed light on these questions: * Is this the same issue as that which https://github.com/ceph/ceph/pull/27929 fixes? * Thus, since https://github.com/ceph/ceph/pull/29115 (the Nautilus backport for that fix) has been merged, but is not yet included in a release, do *Nautilus* users get a fix in the upcoming 14.2.3 release, and once they update, would this bug go away with no further intervention required? * For users on *Luminous*, since https://tracker.ceph.com/issues/39694 (the Luminous version of 38724) says "non-trivial backport", is it fair to say that a fix might still take a while for that release? * Finally, are Luminous users safe from this bug if they keep using, or revert to, FileStore? Thanks in advance for your thoughts! Please keep Erik CC'd on your reply. Cheers, Florian
On Fri, 9 Aug 2019, Florian Haas wrote:
Hi everyone,
it seems there have been several reports in the past related to BlueStore OSDs crashing from unhandled errors in _txc_add_transaction:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-April/034444.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032172.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-December/031960.htm... http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-December/031964.htm...
Bug #38724 tracks this, has been fixed in master with https://github.com/ceph/ceph/pull/27929, and is pending backports (and, I dare say, is *probably* misclassified as being only minor, as this does cause potential data loss as soon as it affects enough OSDs simultaneously):
https://tracker.ceph.com/issues/38724
We just ran into a similar issue with a couple of BlueStore OSDs that we recently added to a Luminous (12.2.12) cluster that was upgraded from Jewel, and hence, still largely runs on FileStore. I say similar because evidently other people reporting this problem have been running into ENOENT (No such file or directory) or ENOTEMPTY (Directory not empty); for us it's interestingly E2BIG (Argument list too long):
{ "op_num": 2, "op_name": "truncate", "collection": "2.293_head", "oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#", "offset": 4457615932 }, That offsize (size) is > 4 GB. BlueStore has a hard limit of 2^32-1 for object sizes (because it uses a uint32_t). This cluster appears to have some ginormous rados objects. Until those are removed, you can't/shouldn't use bluestore. This makes me think we should have scrub issue errors if it encounters rados objects that are bigger than the configured limit. And bluestore should refuse to start if the configured limit is > 4GB. Or something along those lines... sage
So I'm wondering if someone could shed light on these questions:
* Is this the same issue as that which https://github.com/ceph/ceph/pull/27929 fixes?
* Thus, since https://github.com/ceph/ceph/pull/29115 (the Nautilus backport for that fix) has been merged, but is not yet included in a release, do *Nautilus* users get a fix in the upcoming 14.2.3 release, and once they update, would this bug go away with no further intervention required?
* For users on *Luminous*, since https://tracker.ceph.com/issues/39694 (the Luminous version of 38724) says "non-trivial backport", is it fair to say that a fix might still take a while for that release?
* Finally, are Luminous users safe from this bug if they keep using, or revert to, FileStore?
Thanks in advance for your thoughts! Please keep Erik CC'd on your reply.
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Sage! Whoa that was quick. :) On 09/08/2019 16:27, Sage Weil wrote:
{ "op_num": 2, "op_name": "truncate", "collection": "2.293_head", "oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#", "offset": 4457615932 },
That offsize (size) is > 4 GB. BlueStore has a hard limit of 2^32-1 for object sizes (because it uses a uint32_t). This cluster appears to have some ginormous rados objects. Until those are removed, you can't/shouldn't use bluestore.
OK, this is interesting. This is an OpenStack Cinder volumes pool, so all the objects in there belong to RBDs. I couldn't think of any situation in which RBD would create a huge object like that. But, as it happens that PG is currently mapped to a primary OSD that is still on FileStore, so I can do a "find -size +1G" on that mount point, and here's what I get: -rw-r--r-- 1 ceph ceph 4457615932 Mar 29 2018 DIR_3/DIR_9/DIR_6/DIR_C/obj-vS6RN9\uQwvXU9DP__head_DBECC693__2 So, bingo. That's a 4.2GB size file whose size matches that offset exactly. But I'm not familiar with that object name format. How did that object get here? And how do I remove it, considering I seem to be unable to access it? rados -p volumes stat 'obj-vS6RN9\uQwvXU9DP' error stat-ing volumes/obj-vS6RN9\uQwvXU9DP: (2) No such file or directory Or is that file just an artifact that doesn't even map to an object? This is turning out to be a learning experience. :) Thanks again for your help! Cheers, Florian
On Fri, Aug 9, 2019 at 8:04 AM Florian Haas <florian@citynetwork.eu> wrote:
Hi Sage!
Whoa that was quick. :)
On 09/08/2019 16:27, Sage Weil wrote:
{ "op_num": 2, "op_name": "truncate", "collection": "2.293_head", "oid": "#-4:c96337db:::temp_recovering_2.293_11123'6472830_288833_head:head#", "offset": 4457615932 },
That offsize (size) is > 4 GB. BlueStore has a hard limit of 2^32-1 for object sizes (because it uses a uint32_t). This cluster appears to have some ginormous rados objects. Until those are removed, you can't/shouldn't use bluestore.
OK, this is interesting.
This is an OpenStack Cinder volumes pool, so all the objects in there belong to RBDs. I couldn't think of any situation in which RBD would create a huge object like that.
But, as it happens that PG is currently mapped to a primary OSD that is still on FileStore, so I can do a "find -size +1G" on that mount point, and here's what I get:
-rw-r--r-- 1 ceph ceph 4457615932 Mar 29 2018 DIR_3/DIR_9/DIR_6/DIR_C/obj-vS6RN9\uQwvXU9DP__head_DBECC693__2
So, bingo. That's a 4.2GB size file whose size matches that offset exactly.
But I'm not familiar with that object name format. How did that object get here? And how do I remove it, considering I seem to be unable to access it?
rados -p volumes stat 'obj-vS6RN9\uQwvXU9DP' error stat-ing volumes/obj-vS6RN9\uQwvXU9DP: (2) No such file or directory
I believe you need to substitute \u with _
Or is that file just an artifact that doesn't even map to an object?
This is turning out to be a learning experience. :)
Thanks again for your help!
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
And bluestore should refuse to start if the configured limit is > 4GB. Or something along those lines...
Just on this point - Bluestore OSDs will fail to start with an osd_max_object_size >=4GB with a helpful error message about the Bluestore hard limit. I was mildly amused when I discovered that luminous OSDs can start with osd_max_object_size = 4GB - 1 byte, but mimic OSDs require it to be <= 4GB - 2 bytes to start without an error. I haven't checked to see if nautilus OSDs require <= 4GB - 3 bytes yet. Cheers, Tom
Hi Tom, responding back on this briefly so that people are in the loop; I'll have more details in a blog post that I hope to get around to writing. On 12/08/2019 11:34, Thomas Byrne - UKRI STFC wrote:
And bluestore should refuse to start if the configured limit is > 4GB. Or something along those lines...
Just on this point - Bluestore OSDs will fail to start with an osd_max_object_size >=4GB with a helpful error message about the Bluestore hard limit. I was mildly amused when I discovered that luminous OSDs can start with osd_max_object_size = 4GB - 1 byte, but mimic OSDs require it to be <= 4GB - 2 bytes to start without an error. I haven't checked to see if nautilus OSDs require <= 4GB - 3 bytes yet.
Yes but that doesn't help users much for clusters where very large objects already exist. Even in Luminous, osd_max_object_size defaults to 128M, but if an OSD already has objects larger than that, it will still happily start up and serve data with FileStore — and crash any newly added BlueStore OSDs unfortunate enough to be mapped to a PG with one or more objects that are 4GiB or larger. The pending PR to make this a scrub error even on FileStore OSDs mitigates this issue (https://github.com/ceph/ceph/pull/29579), but it'll still cause a somewhat unexpected surprise for people who have just updated to a version including that fix and suddenly see tons of scrub errors — they would be easily forgiven for assuming they've run into a regression that involves false positives on scrub. "Hey, none of these errors were here before the upgrade, surely there's a problem with the software rather than my data!" We've progressed further in the interim and it appears like I can give all-clears on a couple of concerns that we had: 1. It looks like these objects were not created by an RBD going haywire, but by something actually using librados to create them, presumably long before the cluster ever went into production. 2. I am not changing the subject line so I don't mess up people's list archives if their MUA doesn't correctly thread based on In-Reply-To or References, but it's now evident that this is *not* related to bug #38724 but instead really just due to objects being too large for BlueStore, like Sage said in his first reply. Thanks for the answer — by the way I have been imploring all my colleagues to watch your Cephalocon talk,[1] which was excellent. Cheers, Florian [1] https://youtu.be/niFNZN5EKvE
participants (4)
-
Alexandre Marangone
-
Florian Haas
-
Sage Weil
-
Thomas Byrne - UKRI STFC