This is expected when using destroy and re-using the OSD.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at
https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Thu, Sep 5, 2019 at 1:05 PM Brad Hubbard <bhubbard(a)redhat.com> wrote:
>
> +dev(a)ceph.io -ceph-devel(a)vger.kernel.org
>
> On Thu, Sep 5, 2019 at 8:56 PM Ugis <ugis22(a)gmail.com> wrote:
> >
> > Hi,
> >
> > ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus
(stable)
> >
> > Yesterday noticed unexpected behavior, probably bug. It seems ceph
> > wrongly calculates osd size if it is replaced with smaller disk.
> >
> > In detail:
> > Starting point: 1 osd disk had failed, ceph had reballanced and osd
> > was marked down.
> >
> > I did remove failed disk(10TB) and replaced with smaller 6TB.
> > Followed disk replacement instructions here:
> >
https://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-osds/
> >
> > Destroy the OSD first:
> > ceph osd destroy {id} --yes-i-really-mean-it
> > Zap a disk for the new OSD, if the disk was used before for other
> > purposes. It’s not necessary for a new disk:
> > ceph-volume lvm zap /dev/sdX
> > Prepare the disk for replacement by using the previously destroyed OSD id:
> > ceph-volume lvm prepare --osd-id {id} --data /dev/sdX
> > And activate the OSD:
> > ceph-volume lvm activate {id} {fsid}
> > I skipped this as was not clear what fsid was needed(probably ceph
> > cluster fsid} and just started osd
> > systemctl start ceph-osd@29
> >
> > OSD came up and reballance started.
> >
> > After some time ceph started to complain following:
> > # ceph health detail
> > HEALTH_WARN 1 nearfull osd(s); 19 pool(s) nearfull; 10 pgs not
> > deep-scrubbed in time
> > OSD_NEARFULL 1 nearfull osd(s)
> > osd.29 is near full
> >
> > #ceph osd df tree
> > --------------------
> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> > AVAIL %USE VAR PGS STATUS TYPE NAME
> > ...
> > 29 hdd 9.09569 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 981 KiB 4.9 GiB
> > 2.2 TiB 59.75 0.99 590 up osd.29
> >
> > Later I noticed that weight of osd.29 was still 9.09569 as for
> > replaced 10TB disk.
> > I did: ceph osd crush reweight osd.29 5.45789
> > Things got back to normal after reballance.
> >
> > Got impression that ceph did not realize that osd had been replaced
> > with smaller disk. Could that be because I skipped activation step? Or
> > this is a bug.
> >
> > Best regards,
> > Ugis
>
>
>
> --
> Cheers,
> Brad
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io