Re: bluestore disks reading as ATARI/AHDI partitions
So, turns out ceph-volume has two modes: raw and lvm. Rook uses the raw mode, while typical non-Rook clusters are using the lvm mode of ceph-volume. And as the lvm mode doesn't use lsblk to list devices, we're only seeing this in Rook deployed clusters. Is there a way we can fix the raw mode here? Best, Sebastian Am 23.06.21 um 01:57 schrieb Blaine Gardner:
From what I have been reading, it's actually quite easy because AHDI partitions are so easily/loosely defined. I suspect Ubuntu has the kernel built with this enabled to allow for users emulating Atari systems.
See here for reference: https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/1... <https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/12>
I suspect that Ceph's Bluestore layout has a high probability of matching these loose AHDI partitions at least in some cases. It seems very good that Centos 7 isn't built with AHDI support.
Blaine
On Tue, 22 Jun 2021 at 16:53, Steven Ellis <sellis@redhat.com <mailto:sellis@redhat.com>> wrote:
I'm rather puzzled as to why a disk is showing AHDI partition layout. It isn't something that should happen on a modern disk
On Wed, 23 Jun 2021 at 10:07, Blaine Gardner <brgardne@redhat.com <mailto:brgardne@redhat.com>> wrote:
Hi All,
We have had a strange issue crop up in Rook, and I wanted to bring it up for discussion more widely since I think it has the potential to affect cephadm deployments as well as Rook.
Here if you want to look through it: https://github.com/rook/rook/issues/7940#issuecomment-865005220 <https://github.com/rook/rook/issues/7940#issuecomment-865005220>
In Rook, we list disks with `lsblk` to be provisioned. I'm not sure how ceph-volume does this internally or how cephadm does this. In Rook (especially with 500+GB SSDs for some reason), many users are starting to see unexpected partitions appear. One user finally found out that this is because the Ubuntu kernel they're using has support for ATARI (AHDI) partitions. The kernel detects these as partitions, but lsblk does not understand the ATARI type and thus sees them as empty partitions. (It's unclear to me yet if `blkid` is able to see the ATARI type.)
In Rook, this is really only a problem for OSDs that are provisioned on nodes (not a downstream-supported use-case). On PVCs (our downstream use-case), we only run the detection/provisioning step once at first setup, and so I believe Rook's downstream isn't affected. When provisioning OSDs on nodes, disk detection and OSD provisioning both run each time we reconcile. I'm not sure if that is true for cephadm or not.
Additionally, the upstream user reports that ATARI/AHDI disk support is disabled for Centos 7. I hope this means we don't have to worry about this with Centos 8 or any RHEL we ship, but it might be good to confirm this for the RHEL versions we ship to customers.
I'll leave the discussion of Rook-specific ideas for me, Seb, and Travis, but here are the questions I have for what we might need to think about for Ceph more broadly:
Do we need to check Centos/RHEL versions for ATARI/AHDI disk support?
Do we need to communicate to Ceph users that their bluestore OSD disks might appear as ATARI partitions on their host systems?
Do we need to modify ceph-volume to do anything special if it finds ATARI/AHDI partitions?
Are there other considerations I'm missing?
Thanks again! And feel free to chime in with questions if I haven't explained things very well. Blaine
--
Steven Ellis
TECHNICAL PORTFOLIO EVANGELIST - APAC
APAC SENIOR PRINCIPAL PRODUCT MANAGER - OPENSHIFT INFRA & DATA SERVICES
Red Hat Asia Pacific - Auckland NZ <https://www.redhat.com>
E: sellis@redhat.com <mailto:sellis@redhat.com> M: +64 21 321 673 <tel:+64-21-321673>
@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig>
This is buggy AHDI partition detection code in the kernel. I think the only options are (1) disable in the kernel (2) patch the kernel to make the partition detection more strict. From a quick look it appears as though almost no checking of the rootsector is being performed, but I would be nervous changing this code without having some sample atari disk images to test it against. (3) make bluestore's first block somehow work around the atari partition detection. This seems pretty unworkable because it's ignoring the first ~150 bytes and then checking for a few random bits and comparing a few integer values. Maybe rook can avoid the issue by avoiding lsblk, or by ignoring AHDI partitions if they appear? sage On Wed, Jun 23, 2021 at 3:21 AM Sebastian Wagner <sewagner@redhat.com> wrote:
So, turns out ceph-volume has two modes: raw and lvm. Rook uses the raw mode, while typical non-Rook clusters are using the lvm mode of ceph-volume. And as the lvm mode doesn't use lsblk to list devices, we're only seeing this in Rook deployed clusters.
Is there a way we can fix the raw mode here?
Best, Sebastian Am 23.06.21 um 01:57 schrieb Blaine Gardner:
From what I have been reading, it's actually quite easy because AHDI partitions are so easily/loosely defined. I suspect Ubuntu has the kernel built with this enabled to allow for users emulating Atari systems.
See here for reference: https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/1...
I suspect that Ceph's Bluestore layout has a high probability of matching these loose AHDI partitions at least in some cases. It seems very good that Centos 7 isn't built with AHDI support.
Blaine
On Tue, 22 Jun 2021 at 16:53, Steven Ellis <sellis@redhat.com> wrote:
I'm rather puzzled as to why a disk is showing AHDI partition layout. It isn't something that should happen on a modern disk
On Wed, 23 Jun 2021 at 10:07, Blaine Gardner <brgardne@redhat.com> wrote:
Hi All,
We have had a strange issue crop up in Rook, and I wanted to bring it up for discussion more widely since I think it has the potential to affect cephadm deployments as well as Rook.
Here if you want to look through it: https://github.com/rook/rook/issues/7940#issuecomment-865005220
In Rook, we list disks with `lsblk` to be provisioned. I'm not sure how ceph-volume does this internally or how cephadm does this. In Rook (especially with 500+GB SSDs for some reason), many users are starting to see unexpected partitions appear. One user finally found out that this is because the Ubuntu kernel they're using has support for ATARI (AHDI) partitions. The kernel detects these as partitions, but lsblk does not understand the ATARI type and thus sees them as empty partitions. (It's unclear to me yet if `blkid` is able to see the ATARI type.)
In Rook, this is really only a problem for OSDs that are provisioned on nodes (not a downstream-supported use-case). On PVCs (our downstream use-case), we only run the detection/provisioning step once at first setup, and so I believe Rook's downstream isn't affected. When provisioning OSDs on nodes, disk detection and OSD provisioning both run each time we reconcile. I'm not sure if that is true for cephadm or not.
Additionally, the upstream user reports that ATARI/AHDI disk support is disabled for Centos 7. I hope this means we don't have to worry about this with Centos 8 or any RHEL we ship, but it might be good to confirm this for the RHEL versions we ship to customers.
I'll leave the discussion of Rook-specific ideas for me, Seb, and Travis, but here are the questions I have for what we might need to think about for Ceph more broadly:
Do we need to check Centos/RHEL versions for ATARI/AHDI disk support?
Do we need to communicate to Ceph users that their bluestore OSD disks might appear as ATARI partitions on their host systems?
Do we need to modify ceph-volume to do anything special if it finds ATARI/AHDI partitions?
Are there other considerations I'm missing?
Thanks again! And feel free to chime in with questions if I haven't explained things very well. Blaine
--
Steven Ellis
TECHNICAL PORTFOLIO EVANGELIST - APAC
APAC SENIOR PRINCIPAL PRODUCT MANAGER - OPENSHIFT INFRA & DATA SERVICES Red Hat Asia Pacific - Auckland NZ <https://www.redhat.com>
E: sellis@redhat.com <sellis@redhat.com> M: +64 21 321 673 <+64-21-321673> @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig>
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
FWIW, there is an example of an LVM device that reports as an ATARI partition in this Rook issue: https://github.com/rook/rook/issues/2047. I don't think this matters a lot because it is still on LVM, but I don't want to ignore it if it is a useful data point. As Sebastian aptly mentioned, I think that Rook starting to use raw mode instead of lvm mode for OSDs is likely why this issue is popping up now when it hadn't been earlier. To speak to Sage's options: (1) From what I have been able to find out, support for ATARI/AHDI disks isn't something that can be disabled in the kernel the same way a kernel module can. It seems that the kernel is either built with the support enabled or it isn't. This means we just have to suggest that users avoid distros built with the ATARI support. (2) I think this would be the most ideal option, but I'm not sure anyone really wants to do this. XD (3) I'm not sure how much of this we could avoid in bluestore. This seems like it might not be super feasible to me, but I'm certainly no expert. As for Rook, I am hoping to do some investigating to see if `blkid` will tell us about atari partitions where `lsblk` does not. I wonder if it might be just as good an option to have `ceph-volume` check for ATARI partitions when we request it provision a raw device and bail out if it detects one. Blaine
I think we can simply ignore this partition type during the prepare job execution in Rook, right? Looks like the simplest solution to me. Thanks! ––––––––– Sébastien Han Senior Principal Software Engineer, Storage Architect "Always give 100%. Unless you're giving blood." On Wed, Jun 23, 2021 at 7:48 PM <blaine.gardner@redhat.com> wrote:
FWIW, there is an example of an LVM device that reports as an ATARI partition in this Rook issue: https://github.com/rook/rook/issues/2047. I don't think this matters a lot because it is still on LVM, but I don't want to ignore it if it is a useful data point. As Sebastian aptly mentioned, I think that Rook starting to use raw mode instead of lvm mode for OSDs is likely why this issue is popping up now when it hadn't been earlier.
To speak to Sage's options: (1) From what I have been able to find out, support for ATARI/AHDI disks isn't something that can be disabled in the kernel the same way a kernel module can. It seems that the kernel is either built with the support enabled or it isn't. This means we just have to suggest that users avoid distros built with the ATARI support. (2) I think this would be the most ideal option, but I'm not sure anyone really wants to do this. XD (3) I'm not sure how much of this we could avoid in bluestore. This seems like it might not be super feasible to me, but I'm certainly no expert.
As for Rook, I am hoping to do some investigating to see if `blkid` will tell us about atari partitions where `lsblk` does not.
I wonder if it might be just as good an option to have `ceph-volume` check for ATARI partitions when we request it provision a raw device and bail out if it detects one.
Blaine _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
I think you are right. From the research I did yesterday, the only tool that reliably identifies atari partitions is `parted`. I wasn't able to find a way to format parted's output as JSON or to output just the partition type with no other info. We will have to do a string search for atari. But that is just details. I'll get a fix in for that soon into Rook. Blaine On Thu, 24 Jun 2021 at 01:10, Sebastien Han <shan@redhat.com> wrote:
I think we can simply ignore this partition type during the prepare job execution in Rook, right? Looks like the simplest solution to me. Thanks! ––––––––– Sébastien Han Senior Principal Software Engineer, Storage Architect
"Always give 100%. Unless you're giving blood."
On Wed, Jun 23, 2021 at 7:48 PM <blaine.gardner@redhat.com> wrote:
FWIW, there is an example of an LVM device that reports as an ATARI partition in this Rook issue: https://github.com/rook/rook/issues/2047. I don't think this matters a lot because it is still on LVM, but I don't want to ignore it if it is a useful data point. As Sebastian aptly mentioned, I think that Rook starting to use raw mode instead of lvm mode for OSDs is likely why this issue is popping up now when it hadn't been earlier.
To speak to Sage's options: (1) From what I have been able to find out, support for ATARI/AHDI disks isn't something that can be disabled in the kernel the same way a kernel module can. It seems that the kernel is either built with the support enabled or it isn't. This means we just have to suggest that users avoid distros built with the ATARI support. (2) I think this would be the most ideal option, but I'm not sure anyone really wants to do this. XD (3) I'm not sure how much of this we could avoid in bluestore. This seems like it might not be super feasible to me, but I'm certainly no expert.
As for Rook, I am hoping to do some investigating to see if `blkid` will tell us about atari partitions where `lsblk` does not.
I wonder if it might be just as good an option to have `ceph-volume` check for ATARI partitions when we request it provision a raw device and bail out if it detects one.
Blaine _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (5)
-
Blaine Gardner
-
blaine.gardner＠redhat.com
-
Sage Weil
-
Sebastian Wagner
-
Sebastien Han