11 Sep
2019
11 Sep
'19
5:38 p.m.
Hello,
While working on CephFS Quick Start guide[1], the major issue that I
came across was choosing the value for pg_num for the pools that will
serve CephFS. I've tried the values from 4 to 128 for both data and
metadata pools and have always got "undersized+peered" instead of
"active+clean". Copying pg_num values from the cluster setup by
vstart.sh (8 for data and 16 for metadata pools) gave me the same
result.
About the cluster: I had a single node running Fedora 29 with 1 MON, 1
MGR, 1 MDS and 3 OSDs each with a disk size of 10 GB. Thinking that
disk size might have a role to play, I changed the number of OSDs to 2
each with 20 GB disks and later with 50 GB disks but neither helped. I
used dnf to install ceph and ceph-deploy to setup the cluster.
I've copied the the cluster status after every attempt here[2] in case
that helps. Any suggestions about pg_num values I should choose and on
the pg_num values that would be nice for a user looking forward to get
quickly started with CephFS?
[1] https://docs.ceph.com/docs/master/start/quick-cephfs/
[2] https://paste.fedoraproject.org/paste/Q-WH8VWtwu6JwF7eW2JmnA
Thanks,
- Rishabh
11 Sep
11 Sep
5:43 p.m.
On Wed, Sep 11, 2019 at 10:38 AM Rishabh Dave <ridave(a)redhat.com> wrote:
Hello,
While working on CephFS Quick Start guide[1], the major issue that I
came across was choosing the value for pg_num for the pools that will
serve CephFS. I've tried the values from 4 to 128 for both data and
metadata pools and have always got "undersized+peered" instead of
"active+clean". Copying pg_num values from the cluster setup by
vstart.sh (8 for data and 16 for metadata pools) gave me the same
result.
About the cluster: I had a single node running Fedora 29 with 1 MON, 1
MGR, 1 MDS and 3 OSDs each with a disk size of 10 GB. Thinking that
disk size might have a role to play, I changed the number of OSDs to 2
each with 20 GB disks and later with 50 GB disks but neither helped. I
used dnf to install ceph and ceph-deploy to setup the cluster.
I've copied the the cluster status after every attempt here[2] in case
that helps. Any suggestions about pg_num values I should choose and on
the pg_num values that would be nice for a user looking forward to get
quickly started with CephFS?
Why not recommend this in quick-start for master or from nautilus stable?
https://ceph.com/rados/new-in-nautilus-pg-merging-and-autotuning/
[1] https://docs.ceph.com/docs/master/start/quick-cephfs/
[2] https://paste.fedoraproject.org/paste/Q-WH8VWtwu6JwF7eW2JmnA
Thanks,
- Rishabh
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
16 Sep
16 Sep
12:32 p.m.
On Wed, 11 Sep 2019 at 23:13, Vasu Kulkarni <vakulkar(a)redhat.com> wrote:
On Wed, Sep 11, 2019 at 10:38 AM Rishabh Dave <ridave(a)redhat.com> wrote:
Although, I could get "active+clean" for all PGs but not HEALTH_OK,
it'll mention be great to mention this anyway. Thanks for pointing
out.
Hello,
While working on CephFS Quick Start guide[1], the major issue that I
came across was choosing the value for pg_num for the pools that will
serve CephFS. I've tried the values from 4 to 128 for both data and
metadata pools and have always got "undersized+peered" instead of
"active+clean". Copying pg_num values from the cluster setup by
vstart.sh (8 for data and 16 for metadata pools) gave me the same
result.
About the cluster: I had a single node running Fedora 29 with 1 MON, 1
MGR, 1 MDS and 3 OSDs each with a disk size of 10 GB. Thinking that
disk size might have a role to play, I changed the number of OSDs to 2
each with 20 GB disks and later with 50 GB disks but neither helped. I
used dnf to install ceph and ceph-deploy to setup the cluster.
I've copied the the cluster status after every attempt here[2] in case
that helps. Any suggestions about pg_num values I should choose and on
the pg_num values that would be nice for a user looking forward to get
quickly started with CephFS?
Why not recommend this in quick-start for master or from nautilus stable?
https://ceph.com/rados/new-in-nautilus-pg-merging-and-autotuning/
11 Sep
11 Sep
6:01 p.m.
On Wed, 11 Sep 2019, Rishabh Dave wrote:
Hello,
While working on CephFS Quick Start guide[1], the major issue that I
came across was choosing the value for pg_num for the pools that will
serve CephFS. I've tried the values from 4 to 128 for both data and
metadata pools and have always got "undersized+peered" instead of
"active+clean". Copying pg_num values from the cluster setup by
vstart.sh (8 for data and 16 for metadata pools) gave me the same
result.
About the cluster: I had a single node running Fedora 29 with 1 MON, 1
MGR, 1 MDS and 3 OSDs each with a disk size of 10 GB. Thinking that
This is unrelated to the PGs or the capacity--the problem is that you have
a single node, and the default CRUSH rule replicates across hosts.
That's why your pools are unhealthy.
You can fix this by creating a new crush rule with 'osd' instead of
'host' as the failure domain, and then setting your pool(s) to use that
rule.
osd crush rule create-replicated <name> <root> create crush rule <name>
for replicated pool to
<type> {<class>} start from <root>,
replicate across buckets of
type <type>, use devices of type
<class> (ssd
or hdd)
osd pool set <poolname> crush_rule <rule-name>
sage
> disk size might have a role to play, I changed the number of OSDs to 2
> each with 20 GB disks and later with 50 GB disks but neither helped. I
> used dnf to install ceph and ceph-deploy to setup the cluster.
>
> I've copied the the cluster status after every attempt here[2] in case
> that helps. Any suggestions about pg_num values I should choose and on
> the pg_num values that would be nice for a user looking forward to get
> quickly started with CephFS?
>
> [1] https://docs.ceph.com/docs/master/start/quick-cephfs/
> [2] https://paste.fedoraproject.org/paste/Q-WH8VWtwu6JwF7eW2JmnA
>
> Thanks,
> - Rishabh
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
>
>
13 Sep
13 Sep
4:57 p.m.
On Wed, 11 Sep 2019 at 23:31, Sage Weil <sage(a)newdream.net> wrote:
On Wed, 11 Sep 2019, Rishabh Dave wrote:
Both, setting the new crush rule to pools and creating OSDs on
separate nodes worked. Thanks!
However, in both cases the PG status wasn't "active+clean". It was "54
active+undersized" and "10 active+undersized+degraded". IMO, it would
ideal for CephFS Quick Start guide to lead to "active+clean". Is there
anything more that can be done and would be suitable for the guide?
Hello,
While working on CephFS Quick Start guide[1], the major issue that I
came across was choosing the value for pg_num for the pools that will
serve CephFS. I've tried the values from 4 to 128 for both data and
metadata pools and have always got "undersized+peered" instead of
"active+clean". Copying pg_num values from the cluster setup by
vstart.sh (8 for data and 16 for metadata pools) gave me the same
result.
About the cluster: I had a single node running Fedora 29 with 1 MON, 1
MGR, 1 MDS and 3 OSDs each with a disk size of 10 GB. Thinking that
This is unrelated to the PGs or the capacity--the problem is that you have
a single node, and the default CRUSH rule replicates across hosts.
That's why your pools are unhealthy.
You can fix this by creating a new crush rule with 'osd' instead of
'host' as the failure domain, and then setting your pool(s) to use that
rule.
osd crush rule create-replicated <name> <root> create crush rule
<name> for replicated pool to
<type> {<class>} start from <root>,
replicate across buckets of
type <type>, use devices of type
<class> (ssd
or hdd)
osd pool set <poolname> crush_rule <rule-name>
sage
7:16 p.m.
Hi,
if you have the proper setup you should always reach active+clean for all your PGs,
- Single node with 2 OSDs: Rule replicates across OSD, set size=2 and min_size=1 on your
pool
- Single node with 3 OSDs: Rule replicates across OSD (default will be size=3 min_size=2
on your pool)
- Multiple nodes with 2 nodes: Rule replicates across HOST, set size=2 and min_size=1 on
your pool
- Multiple nodes with3 nodes: Rule replicates across HOST (default will be size=3
min_size=2 on your pool)
Tip: For single node deployment set osd_crush_chooseleaf_type = 0 in your configuration
file [global] section before you deploy your MONs and OSDs and it will create the correct
CRUSH rule
Regards
JC
On Sep 13, 2019, at 09:57, Rishabh Dave
<ridave(a)redhat.com> wrote:
On Wed, 11 Sep 2019 at 23:31, Sage Weil <sage(a)newdream.net
<mailto:sage@newdream.net>> wrote:
On Wed, 11 Sep 2019, Rishabh Dave wrote:
Both, setting the new crush rule to pools and creating OSDs on
separate nodes worked. Thanks!
However, in both cases the PG status wasn't "active+clean". It was
"54
active+undersized" and "10 active+undersized+degraded". IMO, it would
ideal for CephFS Quick Start guide to lead to "active+clean". Is there
anything more that can be done and would be suitable for the guide?
_______________________________________________
Dev mailing list -- dev(a)ceph.io <mailto:dev@ceph.io>
To unsubscribe send an email to dev-leave(a)ceph.io <mailto:dev-leave@ceph.io>
Hello,
While working on CephFS Quick Start guide[1], the major issue that I
came across was choosing the value for pg_num for the pools that will
serve CephFS. I've tried the values from 4 to 128 for both data and
metadata pools and have always got "undersized+peered" instead of
"active+clean". Copying pg_num values from the cluster setup by
vstart.sh (8 for data and 16 for metadata pools) gave me the same
result.
About the cluster: I had a single node running Fedora 29 with 1 MON, 1
MGR, 1 MDS and 3 OSDs each with a disk size of 10 GB. Thinking that
This is unrelated to the PGs or the capacity--the problem is that you have
a single node, and the default CRUSH rule replicates across hosts.
That's why your pools are unhealthy.
You can fix this by creating a new crush rule with 'osd' instead of
'host' as the failure domain, and then setting your pool(s) to use that
rule.
osd crush rule create-replicated <name> <root> create crush rule
<name> for replicated pool to
<type> {<class>} start from <root>,
replicate across buckets of
type <type>, use devices of type
<class> (ssd
or hdd)
osd pool set <poolname> crush_rule <rule-name>
sage
16 Sep
16 Sep
11:59 a.m.
On Sat, 14 Sep 2019 at 00:46, JC Lopez <jelopez(a)redhat.com> wrote:
Hi,
if you have the proper setup you should always reach active+clean for all your PGs,
- Single node with 2 OSDs: Rule replicates across OSD, set size=2 and min_size=1 on your
pool
- Single node with 3 OSDs: Rule replicates across OSD (default will be size=3 min_size=2
on your pool)
- Multiple nodes with 2 nodes: Rule replicates across HOST, set size=2 and min_size=1 on
your pool
- Multiple nodes with3 nodes: Rule replicates across HOST (default will be size=3
min_size=2 on your pool)
Tip: For single node deployment set osd_crush_chooseleaf_type = 0 in your configuration
file [global] section before you deploy your MONs and OSDs and it will create the correct
CRUSH rule
I get a perfect cluster status now. I can see HEALTH_OK as well
"active+clean" for all PGs. Thank you!
1716
days inactive
1721
days old
6 comments
4 participants
participants (4)
-
JC Lopez -
Rishabh Dave -
Sage Weil -
Vasu Kulkarni