The fix was to upgrade podman and the issue went away on any containers
restarted, so I'll just do a rolling reboot to clear that issue. Probably
upgrade ceph to 15.2.9 at the same time.
It's a bit of a dev/proof of concept cluster currently, bit it does mean
we're going to need to workout which distro to go with going forward and
which tool box to use....
Everyone here is too used to Spectrum Scale or Lustre and this is the first
time I've really even played with ceph.
Peter.
On Sun, 28 Feb 2021, 18:13 David Orman, <ormandj(a)corenode.com> wrote:
Perhaps just swap out your hosts one at a time with a
different
distribution that's more current. We also use podman from the Kubic
project instead of the OS-provided version. Just make sure to backup
package files when you install versions from there, as they wipe their
repos of the old version when new versions come out, leaving you with
little ability to roll-back.
That output you see is related to LVM, and will likely go away when
you reboot. We see the same behavior even with Podman 3.0.1, but part
of our setup process involves rebooting all hosts in order to ensure
they behave properly. FWIW, this output doesn't impact anything
negatively, other than being annoying on every ceph command.
I suppose what I'm asking is what this: "start having problems
starting the OSD up" means, specifically, from your initial email.
What behavior do you see? What do logs show? Hopefully that will help
pinpoint the root cause of your problems.
On Sun, Feb 28, 2021 at 4:21 AM Peter Childs <pchilds(a)bcs.org> wrote:
Currently I'm using the default podman that comes with CentOS7 1.6.4
which I
fear is the issue.
/bin/podman: stderr WARNING: The same type, major and minor should not
be used for
multiple devices.
Looks to be part of the issue, and I've heard this is an issue in older
versions of podman.
I can't see a ram issue, or a CPU issue it looks like its probably an
issue
with podman mounting overlays, so maybe upgrading podman past that
available with CentOS 7 is the first plan, shame CentOS 8 is a non-project
now :(
Peter.
On Sat, 27 Feb 2021 at 19:37, David Orman <ormandj(a)corenode.com> wrote:
>
> Podman is fine (preferably 3.0+). What were those variables set to
> before? With most recent distributions and kernels we've not noticed a
> problem with the defaults. Did you notice errors that lead to you
> changing them? We have many clusters of 21 nodes, 24 HDDs each,
> multiple NVMEs serving as WAL/DB which were on 15.2.7 and prior, but
> now all are 15.2.9, running in podman 3.0.1 (fixes issues with the 2.2
> series on upgrade). We have less RAM (128G) per node without issues.
>
> On the OSDs that will not start - what error(s) do you see? You can
> inspect the OSDs with "podman logs <id>" if they've started
inside of
> podman but just aren't joining the cluster; if they haven't, then
> looking at the systemctl status for the service or journalctl will
> normally give more insight. Hopefully the root cause of your problems
> can be identified so it can be addressed directly.
>
> On Sat, Feb 27, 2021 at 11:34 AM Peter Childs <pchilds(a)bcs.org> wrote:
> >
> > I'm new to ceph, and I've been trying to set up a new cluster with 16
> > computers with 30 disks each and 6 SSD (plus boot disks), 256G of
memory,
> > IB Networking. (ok its currently 15 but
never mind)
> >
> > When I take them over about 10 OSD's each they start having problems
> > starting the OSD up and I can normally fix this by rebooting them and
it
> > will continue again for a while, and it
is possible to get them up to
the
> > full complement with a bit of poking
around. (Once its working it fne
> > unless you start adding services or moving the OSD's around
> >
> > Is there anything I can change to make it a bit more stable.
> >
> > I've already set
> >
> > fs.aio-max-nr = 1048576
> > kernel.pid_max = 4194303
> > fs.file-max = 500000
> >
> > which made it a bit better, but I feel it could be even better.
> >
> > I'm currently trying to upgrade to 15.2.9 from the default cephadm
version
> > of octopus. The upgrade is going very
very slowly. I'm currently
using
> > podman if that helps, I'm not sure
if docker would be better? (I've
mainly
> > used singularity when I've handled
containers before)
> >
> > Thanks in advance
> >
> > Peter Childs
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io