So, turns out ceph-volume has two modes: raw and lvm. Rook uses the raw
mode,
while typical non-Rook clusters are using the lvm mode of ceph-volume. And
as the lvm mode doesn't use lsblk to list devices, we're only seeing
this in Rook
deployed clusters.
Is there a way we can fix the raw mode here?
Best,
Sebastian
Am 23.06.21 um 01:57 schrieb Blaine Gardner:
From what I have been reading, it's actually quite
easy because AHDI
partitions are so easily/loosely defined. I suspect Ubuntu has the
kernel built with this enabled to allow for users emulating Atari systems.
See here for reference:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/…
<https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1531404/comments/12>
I suspect that Ceph's Bluestore layout has a high probability of
matching these loose AHDI partitions at least in some cases. It seems
very good that Centos 7 isn't built with AHDI support.
Blaine
On Tue, 22 Jun 2021 at 16:53, Steven Ellis <sellis(a)redhat.com
<mailto:sellis@redhat.com>> wrote:
I'm rather puzzled as to why a disk is showing AHDI partition
layout. It isn't something that should happen on a modern disk
On Wed, 23 Jun 2021 at 10:07, Blaine Gardner <brgardne(a)redhat.com
<mailto:brgardne@redhat.com>> wrote:
Hi All,
We have had a strange issue crop up in Rook, and I wanted to
bring it
up for discussion more widely since I think it has the
potential to
affect cephadm deployments as well as Rook.
Here if you want to look through it:
https://github.com/rook/rook/issues/7940#issuecomment-865005220
<https://github.com/rook/rook/issues/7940#issuecomment-865005220>
In Rook, we list disks with `lsblk` to be provisioned. I'm not
sure
how ceph-volume does this internally or how cephadm does this.
In Rook
(especially with 500+GB SSDs for some reason), many users are
starting
to see unexpected partitions appear. One user finally found
out that
this is because the Ubuntu kernel they're using has support
for ATARI
(AHDI) partitions. The kernel detects these as partitions, but
lsblk
does not understand the ATARI type and thus sees them as empty
partitions. (It's unclear to me yet if `blkid` is able to see the
ATARI type.)
In Rook, this is really only a problem for OSDs that are
provisioned
on nodes (not a downstream-supported use-case). On PVCs (our
downstream use-case), we only run the detection/provisioning
step once
at first setup, and so I believe Rook's downstream isn't affected.
When provisioning OSDs on nodes, disk detection and OSD
provisioning
both run each time we reconcile. I'm not sure if that is true for
cephadm or not.
Additionally, the upstream user reports that ATARI/AHDI disk
support
is disabled for Centos 7. I hope this means we don't have to worry
about this with Centos 8 or any RHEL we ship, but it might be
good to
confirm this for the RHEL versions we ship to customers.
I'll leave the discussion of Rook-specific ideas for me, Seb, and
Travis, but here are the questions I have for what we might
need to
think about for Ceph more broadly:
Do we need to check Centos/RHEL versions for ATARI/AHDI disk
support?
Do we need to communicate to Ceph users that their bluestore
OSD disks
might appear as ATARI partitions on their host systems?
Do we need to modify ceph-volume to do anything special if it
finds
ATARI/AHDI partitions?
Are there other considerations I'm missing?
Thanks again! And feel free to chime in with questions if I
haven't
explained things very well.
Blaine
--
Steven Ellis
TECHNICAL PORTFOLIO EVANGELIST - APAC
<https://www.redhat.com>
APAC SENIOR PRINCIPAL PRODUCT MANAGER - OPENSHIFT INFRA & DATA
SERVICES
Red Hat Asia Pacific - Auckland NZ <https://www.redhat.com>
E: sellis(a)redhat.com <mailto:sellis@redhat.com> M: +64 21 321 673
<tel:+64-21-321673>
@RedHat <https://twitter.com/redhat> Red Hat
<https://www.linkedin.com/company/red-hat> Red Hat
<https://www.facebook.com/RedHatInc>
<https://red.ht/sig>
<https://redhat.com/trusted>