Peter,
We're seeing the same issues as you are. We have 2 new hosts Intel(R)
Xeon(R) Gold 6248R CPU @ 3.00GHz w/ 48 cores, 384GB RAM, and 60x 10TB SED
drives and we have tried both 15.2.13 and 16.2.4
Cephadm does NOT properly deploy and activate OSDs on Ubuntu 20.04.2 with
Docker.
Seems to be a bug in Cephadm and a product regression, as we have 4 near
identical nodes on Centos running Nautilus (240 x 10TB SED drives) and had
no problems.
FWIW we had no luck yet with one-by-one OSD daemon additions through ceph
orch either. We also reproduced the issue easily in a virtual lab using
small virtual disks on a single ceph VM with 1 mon.
We are now looking into whether we can get past this with a manual buildout.
If you, or anyone, has hit the same stumbling block and gotten past it, I
would really appreciate some guidance.
Thanks,
Marco
On Thu, May 27, 2021 at 2:23 PM Peter Childs <pchilds(a)bcs.org> wrote:
> In the end it looks like I might be able to get the node up to about 30
> odds before it stops creating any more.
>
> Or more it formats the disks but freezes up starting the daemons.
>
> I suspect I'm missing somthing I can tune to get it working better.
>
> If I could see any error messages that might help, but I'm yet to spit
> anything.
>
> Peter.
>
> On Wed, 26 May 2021, 10:57 Eugen Block, <eblock(a)nde.ag> wrote:
>
> > > If I add the osd daemons one at a time with
> > >
> > > ceph orch daemon add osd drywood12:/dev/sda
> > >
> > > It does actually work,
> >
> > Great!
> >
> > > I suspect what's happening is when my rule for creating osds run and
> > > creates them all-at-once it ties the orch it overloads cephadm and it
> > can't
> > > cope.
> >
> > It's possible, I guess.
> >
> > > I suspect what I might need to do at least to work around the issue is
> > set
> > > "limit:" and bring it up until it stops working.
> >
> > It's worth a try, yes, although the docs state you should try to avoid
> > it, it's possible that it doesn't work properly, in that case create a
> > bug report. ;-)
> >
> > > I did work out how to get ceph-volume to nearly work manually.
> > >
> > > cephadm shell
> > > ceph auth get client.bootstrap-osd -o
> > > /var/lib/ceph/bootstrap-osd/ceph.keyring
> > > ceph-volume lvm create --data /dev/sda --dmcrypt
> > >
> > > but given I've now got "add osd" to work, I suspect I just need to fine
> > > tune my osd creation rules, so it does not try and create too many osds
> > on
> > > the same node at the same time.
> >
> > I agree, no need to do it manually if there is an automated way,
> > especially if you're trying to bring up dozens of OSDs.
> >
> >
> > Zitat von Peter Childs <pchilds(a)bcs.org>:
> >
> > > After a bit of messing around. I managed to get it somewhat working.
> > >
> > > If I add the osd daemons one at a time with
> > >
> > > ceph orch daemon add osd drywood12:/dev/sda
> > >
> > > It does actually work,
> > >
> > > I suspect what's happening is when my rule for creating osds run and
> > > creates them all-at-once it ties the orch it overloads cephadm and it
> > can't
> > > cope.
> > >
> > > service_type: osd
> > > service_name: osd.drywood-disks
> > > placement:
> > > host_pattern: 'drywood*'
> > > spec:
> > > data_devices:
> > > size: "7TB:"
> > > objectstore: bluestore
> > >
> > > I suspect what I might need to do at least to work around the issue is
> > set
> > > "limit:" and bring it up until it stops working.
> > >
> > > I did work out how to get ceph-volume to nearly work manually.
> > >
> > > cephadm shell
> > > ceph auth get client.bootstrap-osd -o
> > > /var/lib/ceph/bootstrap-osd/ceph.keyring
> > > ceph-volume lvm create --data /dev/sda --dmcrypt
> > >
> > > but given I've now got "add osd" to work, I suspect I just need to fine
> > > tune my osd creation rules, so it does not try and create too many osds
> > on
> > > the same node at the same time.
> > >
> > >
> > >
> > > On Wed, 26 May 2021 at 08:25, Eugen Block <eblock(a)nde.ag> wrote:
> > >
> > >> Hi,
> > >>
> > >> I believe your current issue is due to a missing keyring for
> > >> client.bootstrap-osd on the OSD node. But even after fixing that
> > >> you'll probably still won't be able to deploy an OSD manually with
> > >> ceph-volume because 'ceph-volume activate' is not supported with
> > >> cephadm [1]. I just tried that in a virtual environment, it fails when
> > >> activating the systemd-unit:
> > >>
> > >> ---snip---
> > >> [2021-05-26 06:47:16,677][ceph_volume.process][INFO ] Running
> > >> command: /usr/bin/systemctl enable
> > >> ceph-volume@lvm-8-1a8fc8ae-8f4c-4f91-b044-d5636bb52456
> > >> [2021-05-26 06:47:16,692][ceph_volume.process][INFO ] stderr Failed
> > >> to connect to bus: No such file or directory
> > >> [2021-05-26 06:47:16,693][ceph_volume.devices.lvm.create][ERROR ] lvm
> > >> activate was unable to complete, while creating the OSD
> > >> Traceback (most recent call last):
> > >> File
> > >> "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py",
> > >> line 32, in create
> > >> Activate([]).activate(args)
> > >> File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py",
> > >> line 16, in is_root
> > >> return func(*a, **kw)
> > >> File
> > >>
> "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py",
> > >> line
> > >> 294, in activate
> > >> activate_bluestore(lvs, args.no_systemd)
> > >> File
> > >>
> "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py",
> > >> line
> > >> 214, in activate_bluestore
> > >> systemctl.enable_volume(osd_id, osd_fsid, 'lvm')
> > >> File
> > >> "/usr/lib/python3.6/site-packages/ceph_volume/systemd/systemctl.py",
> > >> line 82, in enable_volume
> > >> return enable(volume_unit % (device_type, id_, fsid))
> > >> File
> > >> "/usr/lib/python3.6/site-packages/ceph_volume/systemd/systemctl.py",
> > >> line 22, in enable
> > >> process.run(['systemctl', 'enable', unit])
> > >> File "/usr/lib/python3.6/site-packages/ceph_volume/process.py",
> > >> line 153, in run
> > >> raise RuntimeError(msg)
> > >> RuntimeError: command returned non-zero exit status: 1
> > >> [2021-05-26 06:47:16,694][ceph_volume.devices.lvm.create][INFO ] will
> > >> rollback OSD ID creation
> > >> [2021-05-26 06:47:16,697][ceph_volume.process][INFO ] Running
> > >> command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd
> > >> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.8
> > >> --yes-i-really-mean-it
> > >> [2021-05-26 06:47:17,597][ceph_volume.process][INFO ] stderr purged
> > osd.8
> > >> ---snip---
> > >>
> > >> There's a workaround described in [2] that's not really an option for
> > >> dozens of OSDs. I think your best approach is to bring cephadm to
> > >> activate the OSDs for you.
> > >> You wrote you didn't find any helpful error messages, but did cephadm
> > >> even try to deploy OSDs? What does your osd spec file look like? Did
> > >> you explicitly run 'ceph orch apply osd -i specfile.yml'? This should
> > >> trigger cephadm and you should see at least some output like this:
> > >>
> > >> Mai 26 08:21:48 pacific1 conmon[31446]: 2021-05-26T06:21:48.466+0000
> > >> 7effc15ff700 0 log_channel(cephadm) log [INF] : Applying service
> > >> osd.ssd-hdd-mix on host pacific2...
> > >> Mai 26 08:21:49 pacific1 conmon[31009]: cephadm
> > >> 2021-05-26T06:21:48.469611+0000 mgr.pacific1.whndiw (mgr.14166) 1646 :
> > >> cephadm [INF] Applying service osd.ssd-hdd-mix on host pacific2...
> > >>
> > >> Regards,
> > >> Eugen
> > >>
> > >> [1] https://tracker.ceph.com/issues/49159
> > >> [2] https://tracker.ceph.com/issues/46691
> > >>
> > >>
> > >> Zitat von Peter Childs <pchilds(a)bcs.org>:
> > >>
> > >> > Not sure what I'm doing wrong, I suspect its the way I'm running
> > >> > ceph-volume.
> > >> >
> > >> > root@drywood12:~# cephadm ceph-volume lvm create --data /dev/sda
> > >> --dmcrypt
> > >> > Inferring fsid 1518c8e0-bbe4-11eb-9772-001e67dc85ea
> > >> > Using recent ceph image ceph/ceph@sha256
> > >> > :54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949
> > >> > /usr/bin/docker: Running command: /usr/bin/ceph-authtool
> > --gen-print-key
> > >> > /usr/bin/docker: Running command: /usr/bin/ceph-authtool
> > --gen-print-key
> > >> > /usr/bin/docker: --> RuntimeError: No valid ceph configuration file
> > was
> > >> > loaded.
> > >> > Traceback (most recent call last):
> > >> > File "/usr/sbin/cephadm", line 8029, in <module>
> > >> > main()
> > >> > File "/usr/sbin/cephadm", line 8017, in main
> > >> > r = ctx.func(ctx)
> > >> > File "/usr/sbin/cephadm", line 1678, in _infer_fsid
> > >> > return func(ctx)
> > >> > File "/usr/sbin/cephadm", line 1738, in _infer_image
> > >> > return func(ctx)
> > >> > File "/usr/sbin/cephadm", line 4514, in command_ceph_volume
> > >> > out, err, code = call_throws(ctx, c.run_cmd(),
> > verbosity=verbosity)
> > >> > File "/usr/sbin/cephadm", line 1464, in call_throws
> > >> > raise RuntimeError('Failed command: %s' % ' '.join(command))
> > >> > RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host
> > >> > --net=host --entrypoint /usr/sbin/ceph-volume --privileged
> > >> --group-add=disk
> > >> > --init -e CONTAINER_IMAGE=ceph/ceph@sha256
> :54e95ae1e11404157d7b329d0t
> > >> >
> > >> > root@drywood12:~# cephadm shell
> > >> > Inferring fsid 1518c8e0-bbe4-11eb-9772-001e67dc85ea
> > >> > Inferring config
> > >> >
> > /var/lib/ceph/1518c8e0-bbe4-11eb-9772-001e67dc85ea/mon.drywood12/config
> > >> > Using recent ceph image ceph/ceph@sha256
> > >> > :54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949
> > >> > root@drywood12:/# ceph-volume lvm create --data /dev/sda --dmcrypt
> > >> > Running command: /usr/bin/ceph-authtool --gen-print-key
> > >> > Running command: /usr/bin/ceph-authtool --gen-print-key
> > >> > Running command: /usr/bin/ceph --cluster ceph --name
> > client.bootstrap-osd
> > >> > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
> > >> > 70054a5c-c176-463a-a0ac-b44c5db0987c
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable
> to
> > >> find
> > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such
> > file
> > >> or
> > >> > directory
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1
> > >> > AuthRegistry(0x7fdef405b378) no keyring found at
> > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable
> to
> > >> find
> > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such
> > file
> > >> or
> > >> > directory
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1
> > >> > AuthRegistry(0x7fdef405ef20) no keyring found at
> > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable
> to
> > >> find
> > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such
> > file
> > >> or
> > >> > directory
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1
> > >> > AuthRegistry(0x7fdef8f0bea0) no keyring found at
> > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef2d9d700 -1
> > monclient(hunting):
> > >> > handle_auth_bad_method server allowed_methods [2] but i only support
> > [1]
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef259c700 -1
> > monclient(hunting):
> > >> > handle_auth_bad_method server allowed_methods [2] but i only support
> > [1]
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef1d9b700 -1
> > monclient(hunting):
> > >> > handle_auth_bad_method server allowed_methods [2] but i only support
> > [1]
> > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 monclient:
> > >> > authenticate NOTE: no keyring found; disabled cephx authentication
> > >> > stderr: [errno 13] RADOS permission denied (error connecting to the
> > >> > cluster)
> > >> > --> RuntimeError: Unable to create a new OSD id
> > >> > root@drywood12:/# lsblk /dev/sda
> > >> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > >> > sda 8:0 0 7.3T 0 disk
> > >> >
> > >> > As far as I can see cephadm gets a little further than this as the
> > disks
> > >> > have lvm volumes on them just the osd's daemons are not created or
> > >> started.
> > >> > So maybe I'm invoking ceph-volume incorrectly.
> > >> >
> > >> >
> > >> > On Tue, 25 May 2021 at 06:57, Peter Childs <pchilds(a)bcs.org> wrote:
> > >> >
> > >> >>
> > >> >>
> > >> >> On Mon, 24 May 2021, 21:08 Marc, <Marc(a)f1-outsourcing.eu> wrote:
> > >> >>
> > >> >>> >
> > >> >>> > I'm attempting to use cephadm and Pacific, currently on debian
> > >> buster,
> > >> >>> > mostly because centos7 ain't supported any more and cenotos8
> ain't
> > >> >>> > support
> > >> >>> > by some of my hardware.
> > >> >>>
> > >> >>> Who says centos7 is not supported any more? Afaik centos7/el7 is
> > being
> > >> >>> supported till its EOL 2024. By then maybe a good alternative for
> > >> >>> el8/stream has surfaced.
> > >> >>>
> > >> >>
> > >> >> Not supported by ceph Pacific, it's our os of choice otherwise.
> > >> >>
> > >> >> My testing says the version available of podman, docker and
> python3,
> > do
> > >> >> not work with Pacific.
> > >> >>
> > >> >> Given I've needed to upgrade docker on buster can we please have a
> > list
> > >> of
> > >> >> versions that work with cephadm, maybe even have cephadm say no,
> > please
> > >> >> upgrade unless your running the right version or better.
> > >> >>
> > >> >>
> > >> >>
> > >> >>> > Anyway I have a few nodes with 59x 7.2TB disks but for some
> reason
> > >> the
> > >> >>> > osd
> > >> >>> > daemons don't start, the disks get formatted and the osd are
> > created
> > >> but
> > >> >>> > the daemons never come up.
> > >> >>>
> > >> >>> what if you try with
> > >> >>> ceph-volume lvm create --data /dev/sdi --dmcrypt ?
> > >> >>>
> > >> >>
> > >> >> I'll have a go.
> > >> >>
> > >> >>
> > >> >>> > They are probably the wrong spec for ceph (48gb of memory and
> > only 4
> > >> >>> > cores)
> > >> >>>
> > >> >>> You can always start with just configuring a few disks per node.
> > That
> > >> >>> should always work.
> > >> >>>
> > >> >>
> > >> >> That was my thought too.
> > >> >>
> > >> >> Thanks
> > >> >>
> > >> >> Peter
> > >> >>
> > >> >>
> > >> >>> > but I was expecting them to start and be either dirt slow or
> crash
> > >> >>> > later,
> > >> >>> > anyway I've got upto 30 of them, so I was hoping on getting at
> > least
> > >> get
> > >> >>> > 6PB of raw storage out of them.
> > >> >>> >
> > >> >>> > As yet I've not spotted any helpful error messages.
> > >> >>> >
> > >> >>> _______________________________________________
> > >> >>> ceph-users mailing list -- ceph-users(a)ceph.io
> > >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > >> >>>
> > >> >>
> > >> > _______________________________________________
> > >> > ceph-users mailing list -- ceph-users(a)ceph.io
> > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > >>
> > >>
> > >> _______________________________________________
> > >> ceph-users mailing list -- ceph-users(a)ceph.io
> > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > >>
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
Hi,
Any way to clean up large-omap in the index pool?
PG deep_scrub didn't help.
I know how to clean in the log pool, but no idea in the index pool :/
It's an octopus deployment 15.2.10.
Thank you
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Dear all,
ceph-mgr-dashboard-15.2.13-0.el7.noarch contains three rpm dependencies
that cannot be resolved here (not part of CentOS & EPEL 7):
python3-cherrypy
python3-routes
python3-jwt
Does anybody know where they are expected to come from?
Thanks,
Andreas
--
| Andreas Haupt | E-Mail: andreas.haupt(a)desy.de
| DESY Zeuthen | WWW: http://www-zeuthen.desy.de/~ahaupt
| Platanenallee 6 | Phone: +49/33762/7-7359
| D-15738 Zeuthen | Fax: +49/33762/7-7216
Hi all,
I want to ask your opinion which Ceph deploy version is better: using
cephadm(docker) or installing from packages?
Cephadm brings lots of convenience: easy upgrade (which sometimes stucks
but still), easy add new OSDs, ability to make placement policies etc.
In contrary I seem to lose some transparency on how Ceph works as well
as ability to apply fine-grained control, namely setting up daemon
configs myself. Even running ceph-volume lvm list to see which osd uses
a particular block devices requires entering a docker container.
So which option is better to use in production cluster?
Hello.
I have virtualization env and I'm looking new SSD for HDD replacement.
What are the best Performance / Price SSDs in the market right now?
I'm looking 1TB, 512GB, 480GB, 256GB, 240GB.
Is there a SSD recommendation list for ceph?
I’m trying to understand this situation:
ceph health detail
HEALTH_WARN Reduced data availability: 33 pgs inactive
[WRN] PG_AVAILABILITY: Reduced data availability: 33 pgs inactive
pg 1.0 is stuck inactive for 20h, current state unknown, last acting []
pg 2.0 is stuck inactive for 20h, current state unknown, last acting []
pg 2.1 is stuck inactive for 20h, current state unknown, last acting []
pg 2.2 is stuck inactive for 20h, current state unknown, last acting []
pg 2.3 is stuck inactive for 20h, current state unknown, last acting []
pg 2.4 is stuck inactive for 20h, current state unknown, last acting []
pg 2.5 is stuck inactive for 20h, current state unknown, last acting []
pg 2.6 is stuck inactive for 20h, current state unknown, last acting []
pg 2.7 is stuck inactive for 20h, current state unknown, last acting []
pg 2.8 is stuck inactive for 20h, current state unknown, last acting []
pg 2.9 is stuck inactive for 20h, current state unknown, last acting []
pg 2.a is stuck inactive for 20h, current state unknown, last acting []
pg 2.b is stuck inactive for 20h, current state unknown, last acting []
pg 2.c is stuck inactive for 20h, current state unknown, last acting []
pg 2.d is stuck inactive for 20h, current state unknown, last acting []
pg 2.e is stuck inactive for 20h, current state unknown, last acting []
pg 2.f is stuck inactive for 20h, current state unknown, last acting []
pg 2.10 is stuck inactive for 20h, current state unknown, last acting []
pg 2.11 is stuck inactive for 20h, current state unknown, last acting []
pg 2.12 is stuck inactive for 20h, current state unknown, last acting []
pg 2.13 is stuck inactive for 20h, current state unknown, last acting []
pg 2.14 is stuck inactive for 20h, current state unknown, last acting []
pg 2.15 is stuck inactive for 20h, current state unknown, last acting []
pg 2.16 is stuck inactive for 20h, current state unknown, last acting []
pg 2.17 is stuck inactive for 20h, current state unknown, last acting []
pg 2.18 is stuck inactive for 20h, current state unknown, last acting []
pg 2.19 is stuck inactive for 20h, current state unknown, last acting []
pg 2.1a is stuck inactive for 20h, current state unknown, last acting []
pg 2.1b is stuck inactive for 20h, current state unknown, last acting []
pg 2.1c is stuck inactive for 20h, current state unknown, last acting []
pg 2.1d is stuck inactive for 20h, current state unknown, last acting []
pg 2.1e is stuck inactive for 20h, current state unknown, last acting []
pg 2.1f is stuck inactive for 20h, current state unknown, last acting []
[ceph: root@cn01 /]# date
Sat May 29 01:28:37 UTC 2021
[ceph: root@cn01 /]# ceph pg dump_stuck inactive
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY
2.1f unknown [] -1 [] -1
2.1e unknown [] -1 [] -1
2.1d unknown [] -1 [] -1
2.1c unknown [] -1 [] -1
2.1b unknown [] -1 [] -1
2.1a unknown [] -1 [] -1
2.19 unknown [] -1 [] -1
2.18 unknown [] -1 [] -1
2.17 unknown [] -1 [] -1
2.16 unknown [] -1 [] -1
2.15 unknown [] -1 [] -1
2.14 unknown [] -1 [] -1
2.13 unknown [] -1 [] -1
2.12 unknown [] -1 [] -1
2.11 unknown [] -1 [] -1
2.10 unknown [] -1 [] -1
2.f unknown [] -1 [] -1
2.9 unknown [] -1 [] -1
2.b unknown [] -1 [] -1
2.c unknown [] -1 [] -1
2.e unknown [] -1 [] -1
2.a unknown [] -1 [] -1
2.d unknown [] -1 [] -1
2.8 unknown [] -1 [] -1
2.7 unknown [] -1 [] -1
2.6 unknown [] -1 [] -1
2.5 unknown [] -1 [] -1
2.0 unknown [] -1 [] -1
1.0 unknown [] -1 [] -1
2.3 unknown [] -1 [] -1
2.1 unknown [] -1 [] -1
2.2 unknown [] -1 [] -1
2.4 unknown [] -1 [] -1
ok
[ceph: root@cn01 /]# ceph pg 2.4 query
Couldn't parse JSON : Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "/usr/bin/ceph", line 1310, in <module>
retval = main()
File "/usr/bin/ceph", line 1230, in main
sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli')
File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 966, in parse_json_funcsigs
raise e
File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 963, in parse_json_funcsigs
overall = json.loads(s)
File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
-jeremy
I’m very new to Ceph so if this question makes no sense, I apologize. Continuing to study but I thought an answer to this question would help me understand Ceph a bit more.
Using cephadm, I set up a cluster. Cephadm automatically creates a pool for Ceph metrics. It looks like one of my ssd osd’s was allocated for the PG. I’d like to understand how to remap this PG so it’s not using the SSD OSDs.
ceph pg map 1.0
osdmap e205 pg 1.0 (1.0) -> up [28,33,10] acting [28,33,10]
OSD 28 is the SSD.
Is this possible? Does this make any sense? I’d like to reserve the SSDs for their own pool.
Thank you!
-jeremy
Is there a way to log or track which cephfs files are being accessed?
This would help us in planning where to place certain datasets based on
popularity, eg on a EC HDD pool or a replicated SSD pool.
I know I can run inotify on the ceph clients, but I was hoping that the
MDS would have a way to log this information centrally.
--Mike
Hoping someone may be able to help point out where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that.
This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
> 15.41G 79% 4.36MB/s 0:56:09 (xfr#23165, ir-chk=4061/27259)
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.17 0.00 1.34 13.23 0.00 85.26
>
> Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz aqu-sz %util
> rbd0 124.00 0.66 0.00 0.00 17.30 5.48 50.00 0.17 0.00 0.00 31.70 3.49 0.00 0.00 0.00 0.00 0.00 0.00 3.39 96.40
Rsync progress and iostat (during the rsync) from the rbd to a local ssd, to remove any bottlenecks doubling back to cephfs.
About 16G in 1h, not exactly blazing, this being 5 of the 7000 directories I'm looking to offload to cephfs.
Currently running 15.2.11, and the host is Ubuntu 20.04 (5.4.0-72-generic) with a single E5-2620, 64GB of memory, and 4x10GbT bond talking to ceph, iperf proves it out.
EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the other 216 being 2TB 7.2K SATA. So there are quite a few spindles in play here.
Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler recommends going to 512, but was hoping to avoid the performance overhead of the PG splits if possible, given perf is bad enough as is.
Examining the main directory structure it looks like there are 7000 files per directory, about 60% of which are <1MiB, and in all totaling nearly 5GiB per directory.
My fstab for this is:
> xfs _netdev,noatime 0 0
I tried to increase the read_ahead_kb to 4M from 128K at /sys/block/rbd0/queue/read_ahead_kb to match the object/stripe size of the EC pool, but that doesn't appear to have had much of an impact.
The only thing I can think of that I could possibly try as a change would be to increase the queue depth in the rbdmap up from 128, so thats my next bullet to fire.
Attaching xfs_info in case there are any useful nuggets:
> meta-data=/dev/rbd0 isize=256 agcount=81, agsize=268435455 blks
> = sectsz=512 attr=2, projid32bit=0
> = crc=0 finobt=0, sparse=0, rmapbt=0
> = reflink=0
> data = bsize=4096 blocks=21483470848, imaxpct=5
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0, ftype=0
> log =internal log bsize=4096 blocks=32768, version=2
> = sectsz=512 sunit=0 blks, lazy-count=0
> realtime =none extsz=4096 blocks=0, rtextents=0
And rbd-info:
> rbd image 'rbd-image-name:
> size 85 TiB in 22282240 objects
> order 22 (4 MiB objects)
> snapshot_count: 0
> id: a09cac2b772af5
> data_pool: rbd-ec82-pool
> block_name_prefix: rbd_data.29.a09cac2b772af5
> format: 2
> features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
> op_features:
> flags:
> create_timestamp: Mon Apr 12 18:44:38 2021
> access_timestamp: Mon Apr 12 18:44:38 2021
> modify_timestamp: Mon Apr 12 18:44:38 2021
Any other ideas or hints are greatly appreciated.
Thanks,
Reed
