23 Jun
2021
23 Jun
'21
9:21 a.m.
So, turns out ceph-volume has two modes: raw and lvm. Rook uses the raw
mode,
while typical non-Rook clusters are using the lvm mode of ceph-volume. And
as the lvm mode doesn't use lsblk to list devices, we're only seeing
this in Rook
deployed clusters.
Is there a way we can fix the raw mode here?
Best,
Sebastian
Am 23.06.21 um 01:57 schrieb Blaine Gardner:
From what I have been reading, it's actually quite
easy because AHDI
partitions are so easily/loosely defined. I suspect Ubuntu has the
kernel built with this enabled to allow for users emulating Atari systems.
See here for reference:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/…
<https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/12>
I suspect that Ceph's Bluestore layout has a high probability of
matching these loose AHDI partitions at least in some cases. It seems
very good that Centos 7 isn't built with AHDI support.
Blaine
On Tue, 22 Jun 2021 at 16:53, Steven Ellis <sellis(a)redhat.com
<mailto:sellis@redhat.com>> wrote:
I'm rather puzzled as to why a disk is showing AHDI partition
layout. It isn't something that should happen on a modern disk
On Wed, 23 Jun 2021 at 10:07, Blaine Gardner <brgardne(a)redhat.com
<mailto:brgardne@redhat.com>> wrote:
Hi All,
We have had a strange issue crop up in Rook, and I wanted to
bring it
up for discussion more widely since I think it has the
potential to
affect cephadm deployments as well as Rook.
Here if you want to look through it:
https://github.com/rook/rook/issues/7940#issuecomment-865005220
<https://github.com/rook/rook/issues/7940#issuecomment-865005220>
In Rook, we list disks with `lsblk` to be provisioned. I'm not
sure
how ceph-volume does this internally or how cephadm does this.
In Rook
(especially with 500+GB SSDs for some reason), many users are
starting
to see unexpected partitions appear. One user finally found
out that
this is because the Ubuntu kernel they're using has support
for ATARI
(AHDI) partitions. The kernel detects these as partitions, but
lsblk
does not understand the ATARI type and thus sees them as empty
partitions. (It's unclear to me yet if `blkid` is able to see the
ATARI type.)
In Rook, this is really only a problem for OSDs that are
provisioned
on nodes (not a downstream-supported use-case). On PVCs (our
downstream use-case), we only run the detection/provisioning
step once
at first setup, and so I believe Rook's downstream isn't affected.
When provisioning OSDs on nodes, disk detection and OSD
provisioning
both run each time we reconcile. I'm not sure if that is true for
cephadm or not.
Additionally, the upstream user reports that ATARI/AHDI disk
support
is disabled for Centos 7. I hope this means we don't have to worry
about this with Centos 8 or any RHEL we ship, but it might be
good to
confirm this for the RHEL versions we ship to customers.
I'll leave the discussion of Rook-specific ideas for me, Seb, and
Travis, but here are the questions I have for what we might
need to
think about for Ceph more broadly:
Do we need to check Centos/RHEL versions for ATARI/AHDI disk
support?
Do we need to communicate to Ceph users that their bluestore
OSD disks
might appear as ATARI partitions on their host systems?
Do we need to modify ceph-volume to do anything special if it
finds
ATARI/AHDI partitions?
Are there other considerations I'm missing?
Thanks again! And feel free to chime in with questions if I
haven't
explained things very well.
Blaine
--
Steven Ellis
TECHNICAL PORTFOLIO EVANGELIST - APAC
<https://www.redhat.com>
APAC SENIOR PRINCIPAL PRODUCT MANAGER - OPENSHIFT INFRA & DATA
SERVICES
Red Hat Asia Pacific - Auckland NZ <https://www.redhat.com>
E: sellis(a)redhat.com <mailto:sellis@redhat.com> M: +64 21 321 673
<tel:+64-21-321673>
@RedHat <https://twitter.com/redhat> Red Hat
<https://www.linkedin.com/company/red-hat> Red Hat
<https://www.facebook.com/RedHatInc>
<https://red.ht/sig>
<https://redhat.com/trusted>
23 Jun
23 Jun
2:20 p.m.
New subject: bluestore disks reading as ATARI/AHDI partitions
This is buggy AHDI partition detection code in the kernel. I think the
only options are
(1) disable in the kernel
(2) patch the kernel to make the partition detection more strict. From a
quick look it appears as though almost no checking of the rootsector is
being performed, but I would be nervous changing this code without having
some sample atari disk images to test it against.
(3) make bluestore's first block somehow work around the atari partition
detection. This seems pretty unworkable because it's ignoring the first
~150 bytes and then checking for a few random bits and comparing a few
integer values.
Maybe rook can avoid the issue by avoiding lsblk, or by ignoring AHDI
partitions if they appear?
sage
On Wed, Jun 23, 2021 at 3:21 AM Sebastian Wagner <sewagner(a)redhat.com>
wrote:
So, turns out ceph-volume has two modes: raw and lvm.
Rook uses the raw
mode,
while typical non-Rook clusters are using the lvm mode of ceph-volume. And
as the lvm mode doesn't use lsblk to list devices, we're only seeing this
in Rook
deployed clusters.
Is there a way we can fix the raw mode here?
Best,
Sebastian
Am 23.06.21 um 01:57 schrieb Blaine Gardner:
From what I have been reading, it's actually quite easy because AHDI
partitions are so easily/loosely defined. I suspect Ubuntu has the kernel
built with this enabled to allow for users emulating Atari systems.
See here for reference:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/…
I suspect that Ceph's Bluestore layout has a high probability of matching
these loose AHDI partitions at least in some cases. It seems very good that
Centos 7 isn't built with AHDI support.
Blaine
On Tue, 22 Jun 2021 at 16:53, Steven Ellis <sellis(a)redhat.com> wrote:
I'm rather puzzled as to why a disk is
showing AHDI partition layout. It
isn't something that should happen on a modern disk
On Wed, 23 Jun 2021 at 10:07, Blaine Gardner <brgardne(a)redhat.com> wrote:
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
Hi All,
We have had a strange issue crop up in Rook, and I wanted to bring it
up for discussion more widely since I think it has the potential to
affect cephadm deployments as well as Rook.
Here if you want to look through it:
https://github.com/rook/rook/issues/7940#issuecomment-865005220
In Rook, we list disks with `lsblk` to be provisioned. I'm not sure
how ceph-volume does this internally or how cephadm does this. In Rook
(especially with 500+GB SSDs for some reason), many users are starting
to see unexpected partitions appear. One user finally found out that
this is because the Ubuntu kernel they're using has support for ATARI
(AHDI) partitions. The kernel detects these as partitions, but lsblk
does not understand the ATARI type and thus sees them as empty
partitions. (It's unclear to me yet if `blkid` is able to see the
ATARI type.)
In Rook, this is really only a problem for OSDs that are provisioned
on nodes (not a downstream-supported use-case). On PVCs (our
downstream use-case), we only run the detection/provisioning step once
at first setup, and so I believe Rook's downstream isn't affected.
When provisioning OSDs on nodes, disk detection and OSD provisioning
both run each time we reconcile. I'm not sure if that is true for
cephadm or not.
Additionally, the upstream user reports that ATARI/AHDI disk support
is disabled for Centos 7. I hope this means we don't have to worry
about this with Centos 8 or any RHEL we ship, but it might be good to
confirm this for the RHEL versions we ship to customers.
I'll leave the discussion of Rook-specific ideas for me, Seb, and
Travis, but here are the questions I have for what we might need to
think about for Ceph more broadly:
Do we need to check Centos/RHEL versions for ATARI/AHDI disk support?
Do we need to communicate to Ceph users that their bluestore OSD disks
might appear as ATARI partitions on their host systems?
Do we need to modify ceph-volume to do anything special if it finds
ATARI/AHDI partitions?
Are there other considerations I'm missing?
Thanks again! And feel free to chime in with questions if I haven't
explained things very well.
Blaine
--
Steven Ellis
TECHNICAL PORTFOLIO EVANGELIST - APAC
<https://www.redhat.com>
APAC SENIOR PRINCIPAL PRODUCT MANAGER - OPENSHIFT INFRA & DATA SERVICES
Red Hat Asia Pacific - Auckland NZ <https://www.redhat.com>
E: sellis(a)redhat.com <sellis(a)redhat.com> M: +64 21 321 673
<+64-21-321673>
@RedHat <https://twitter.com/redhat> Red Hat
<https://www.linkedin.com/company/red-hat> Red Hat
<https://www.facebook.com/RedHatInc>
<https://red.ht/sig>
<https://redhat.com/trusted>
6:47 p.m.
New subject: bluestore disks reading as ATARI/AHDI partitions
FWIW, there is an example of an LVM device that reports as an ATARI partition in this Rook
issue: https://github.com/rook/rook/issues/2047. I don't think this matters a lot
because it is still on LVM, but I don't want to ignore it if it is a useful data
point. As Sebastian aptly mentioned, I think that Rook starting to use raw mode instead of
lvm mode for OSDs is likely why this issue is popping up now when it hadn't been
earlier.
To speak to Sage's options:
(1) From what I have been able to find out, support for ATARI/AHDI disks isn't
something that can be disabled in the kernel the same way a kernel module can. It seems
that the kernel is either built with the support enabled or it isn't. This means we
just have to suggest that users avoid distros built with the ATARI support.
(2) I think this would be the most ideal option, but I'm not sure anyone really wants
to do this. XD
(3) I'm not sure how much of this we could avoid in bluestore. This seems like it
might not be super feasible to me, but I'm certainly no expert.
As for Rook, I am hoping to do some investigating to see if `blkid` will tell us about
atari partitions where `lsblk` does not.
I wonder if it might be just as good an option to have `ceph-volume` check for ATARI
partitions when we request it provision a raw device and bail out if it detects one.
Blaine
24 Jun
24 Jun
8:09 a.m.
New subject: bluestore disks reading as ATARI/AHDI partitions
I think we can simply ignore this partition type during the prepare
job execution in Rook, right? Looks like the simplest solution to me.
Thanks!
–––––––––
Sébastien Han
Senior Principal Software Engineer, Storage Architect
"Always give 100%. Unless you're giving blood."
On Wed, Jun 23, 2021 at 7:48 PM <blaine.gardner(a)redhat.com> wrote:
FWIW, there is an example of an LVM device that reports as an ATARI partition in this
Rook issue: https://github.com/rook/rook/issues/2047. I don't think this matters a lot
because it is still on LVM, but I don't want to ignore it if it is a useful data
point. As Sebastian aptly mentioned, I think that Rook starting to use raw mode instead of
lvm mode for OSDs is likely why this issue is popping up now when it hadn't been
earlier.
To speak to Sage's options:
(1) From what I have been able to find out, support for ATARI/AHDI disks isn't
something that can be disabled in the kernel the same way a kernel module can. It seems
that the kernel is either built with the support enabled or it isn't. This means we
just have to suggest that users avoid distros built with the ATARI support.
(2) I think this would be the most ideal option, but I'm not sure anyone really wants
to do this. XD
(3) I'm not sure how much of this we could avoid in bluestore. This seems like it
might not be super feasible to me, but I'm certainly no expert.
As for Rook, I am hoping to do some investigating to see if `blkid` will tell us about
atari partitions where `lsblk` does not.
I wonder if it might be just as good an option to have `ceph-volume` check for ATARI
partitions when we request it provision a raw device and bail out if it detects one.
Blaine
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
4:46 p.m.
New subject: bluestore disks reading as ATARI/AHDI partitions
I think you are right. From the research I did yesterday, the only
tool that reliably identifies atari partitions is `parted`. I wasn't
able to find a way to format parted's output as JSON or to output just
the partition type with no other info. We will have to do a string
search for atari. But that is just details. I'll get a fix in for that
soon into Rook.
Blaine
On Thu, 24 Jun 2021 at 01:10, Sebastien Han <shan(a)redhat.com> wrote:
I think we can simply ignore this partition type during the prepare
job execution in Rook, right? Looks like the simplest solution to me.
Thanks!
–––––––––
Sébastien Han
Senior Principal Software Engineer, Storage Architect
"Always give 100%. Unless you're giving blood."
On Wed, Jun 23, 2021 at 7:48 PM <blaine.gardner(a)redhat.com> wrote:
FWIW, there is an example of an LVM device that reports as an ATARI partition in this
Rook issue: https://github.com/rook/rook/issues/2047. I don't think this matters a lot
because it is still on LVM, but I don't want to ignore it if it is a useful data
point. As Sebastian aptly mentioned, I think that Rook starting to use raw mode instead of
lvm mode for OSDs is likely why this issue is popping up now when it hadn't been
earlier.
To speak to Sage's options:
(1) From what I have been able to find out, support for ATARI/AHDI disks isn't
something that can be disabled in the kernel the same way a kernel module can. It seems
that the kernel is either built with the support enabled or it isn't. This means we
just have to suggest that users avoid distros built with the ATARI support.
(2) I think this would be the most ideal option, but I'm not sure anyone really wants
to do this. XD
(3) I'm not sure how much of this we could avoid in bluestore. This seems like it
might not be super feasible to me, but I'm certainly no expert.
As for Rook, I am hoping to do some investigating to see if `blkid` will tell us about
atari partitions where `lsblk` does not.
I wonder if it might be just as good an option to have `ceph-volume` check for ATARI
partitions when we request it provision a raw device and bail out if it detects one.
Blaine
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
1059
days inactive
1060
days old
4 comments
5 participants
participants (5)
-
Blaine Gardner -
blaine.gardner＠redhat.com -
Sage Weil -
Sebastian Wagner -
Sebastien Han