Presently I have about 1.2B objects (400M w/3 Replicas) and I'm finding the PG
scrubbing and deep scrubbing are not completing. There is only 1 client accessing the
data, a Samba server. I found large disparities in PG Distribution and Drive utilization.
I enabled pg_autoscaler and found that it was reducing the number of PGs per OSD from 116
to 104.5 at this time, but it wasn't helping with space consumption equalization. I
found loadbalancer and enabled that and it is in the process of evening out. As we were
also having mds crashes even after increasing mds memory, i tried enabling multimds with
rank:2
I currently have only 1 spare and would like to potentially enable the mds component on
the fourth node (no mon present) but am having some difficulty.
Is mon a requirement?
I tried ceph-deploy mds create node4 but am having errors. I tried manually creating the
/var/lib/ceph/mds/node4 directory and the command to create the keyring but still no joy.
What am i missing?
Thanks,
Show replies by date
Please show the exact errors you are getting. You should not need a
mon on the same node to create an mds, though you will want an
/etc/ceph/ceph.conf that points to the mons.
On Fri, Feb 7, 2020 at 2:58 PM <marcopizzolo(a)gmail.com> wrote:
>
> Presently I have about 1.2B objects (400M w/3 Replicas) and I'm finding the PG
scrubbing and deep scrubbing are not completing. There is only 1 client accessing the
data, a Samba server. I found large disparities in PG Distribution and Drive utilization.
I enabled pg_autoscaler and found that it was reducing the number of PGs per OSD from 116
to 104.5 at this time, but it wasn't helping with space consumption equalization. I
found loadbalancer and enabled that and it is in the process of evening out. As we were
also having mds crashes even after increasing mds memory, i tried enabling multimds with
rank:2
>
> I currently have only 1 spare and would like to potentially enable the mds component
on the fourth node (no mon present) but am having some difficulty.
>
> Is mon a requirement?
>
> I tried ceph-deploy mds create node4 but am having errors. I tried manually creating
the /var/lib/ceph/mds/node4 directory and the command to create the keyring but still no
joy.
>
> What am i missing?
>
> Thanks,
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
Thanks Nathan,
Having worked on this a bit since I did make some progress:
[prdceph04][DEBUG ] connected to host: prdceph04
[prdceph04][DEBUG ] detect platform information from remote host
[prdceph04][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.7.1908 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to prdceph04
[prdceph04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[prdceph04][WARNIN] mds keyring does not exist yet, creating one
[prdceph04][DEBUG ] create a keyring file
[prdceph04][DEBUG ] create path if it doesn't exist
[prdceph04][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds
--keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.prdceph04 osd
allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-prdceph04/keyring
[prdceph04][INFO ] Running command: systemctl enable ceph-mds@prdceph04
[prdceph04][WARNIN] Created symlink from
/etc/systemd/system/ceph-mds.target.wants/ceph-mds(a)prdceph04.service to
/usr/lib/systemd/system/ceph-mds@.service.
[prdceph04][INFO ] Running command: systemctl start ceph-mds@prdceph04
[prdceph04][INFO ] Running command: systemctl enable ceph.target
However i don't see it in dashboard or in ceph -s:
health: HEALTH_WARN
1 pools have many more objects per pg than average
Degraded data redundancy: 1351072/1169191146 objects degraded (0.116%), 39 pgs
degraded, 39 pgs undersized
108 pgs not deep-scrubbed in time
12 pgs not scrubbed in time
services:
mon: 3 daemons, quorum prdceph01,prdceph02,prdceph03 (age 2h)
mgr: prdceph01(active, since 2h), standbys: prdceph03, prdceph04, prdceph02
mds: ArchiveRepository:2 {0=prdceph03=up:active,1=prdceph02=up:active} 1 up:standby
osd: 240 osds: 240 up (since 2h), 240 in; 323 remapped pgs
data:
pools: 7 pools, 8383 pgs
objects: 389.73M objects, 460 TiB
usage: 1.4 PiB used, 763 TiB / 2.1 PiB avail
pgs: 1351072/1169191146 objects degraded (0.116%)
11416275/1169191146 objects misplaced (0.976%)
8054 active+clean
242 active+remapped+backfill_wait
42 active+remapped+backfilling
29 active+undersized+degraded+remapped+backfill_wait
10 active+undersized+degraded+remapped+backfilling
6 active+clean+scrubbing+deep
I think i have it working now... I'm sure i missed something early on that would have
made this simpler.
Thanks Nathan all the same.