Unexpected ceph pool creation error with Ceph Quincy
Hi all, I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create". When not passing any pg_num and pgp_num values, I get the following error with Quincy: [root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1 I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed. I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf: [root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8 At least, this is what appears to be used with Pacific. Is this an intended change of behavior? I could not find anything related in the release notes. Best regards, Geert Kloosterman
Hi, I could not confirm this in a virtual lab cluster, also on 17.2.5: host1:~ # ceph osd pool create asdf pool 'asdf' created host1:~ # ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0 So it looks quite similar except the pgp_num value (I can't remember having that modified). This is an upgraded Nautilus cluster. Zitat von Geert Kloosterman <gkloosterman@nvidia.com>:
Hi all,
I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create".
When not passing any pg_num and pgp_num values, I get the following error with Quincy:
[root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1
I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed.
I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf:
[root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8
At least, this is what appears to be used with Pacific.
Is this an intended change of behavior? I could not find anything related in the release notes.
Best regards, Geert Kloosterman _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Thanks Eugen for checking this. I get the same default values as you when I remove the entries from my ceph.conf: [root@gjk-ceph ~]# ceph-conf -D | grep default_pg osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0 However, in my case, the pool creation error remains: [root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1 But I can create the pool when passing the same pg_num and pgp_num values explicity: [root@gjk-ceph ~]# ceph osd pool create asdf 32 0 pool 'asdf' created Does anyone have an idea how I can debug this further? I'm running Ceph on a virtualized Rocky 8.7 test cluster, with Ceph rpms installed from http://download.ceph.com/rpm-quincy/el8/ Best regards, Geert Kloosterman On Wed, 2023-03-15 at 13:42 +0000, Eugen Block wrote:
External email: Use caution opening links or attachments
Hi,
I could not confirm this in a virtual lab cluster, also on 17.2.5:
host1:~ # ceph osd pool create asdf pool 'asdf' created
host1:~ # ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0
So it looks quite similar except the pgp_num value (I can't remember having that modified). This is an upgraded Nautilus cluster.
Zitat von Geert Kloosterman <gkloosterman@nvidia.com>:
Hi all,
I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create".
When not passing any pg_num and pgp_num values, I get the following error with Quincy:
[root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1
I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed.
I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf:
[root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8
At least, this is what appears to be used with Pacific.
Is this an intended change of behavior? I could not find anything related in the release notes.
Best regards, Geert Kloosterman _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Did you ever adjust mon_max_pool_pg_num? Can you check what your current config value is? host1:~ # ceph config get mon mon_max_pool_pg_num 65536 Zitat von Geert Kloosterman <gkloosterman@nvidia.com>:
Hi,
Thanks Eugen for checking this. I get the same default values as you when I remove the entries from my ceph.conf:
[root@gjk-ceph ~]# ceph-conf -D | grep default_pg osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0
However, in my case, the pool creation error remains:
[root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1
But I can create the pool when passing the same pg_num and pgp_num values explicity:
[root@gjk-ceph ~]# ceph osd pool create asdf 32 0 pool 'asdf' created
Does anyone have an idea how I can debug this further?
I'm running Ceph on a virtualized Rocky 8.7 test cluster, with Ceph rpms installed from http://download.ceph.com/rpm-quincy/el8/
Best regards, Geert Kloosterman
On Wed, 2023-03-15 at 13:42 +0000, Eugen Block wrote:
External email: Use caution opening links or attachments
Hi,
I could not confirm this in a virtual lab cluster, also on 17.2.5:
host1:~ # ceph osd pool create asdf pool 'asdf' created
host1:~ # ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0
So it looks quite similar except the pgp_num value (I can't remember having that modified). This is an upgraded Nautilus cluster.
Zitat von Geert Kloosterman <gkloosterman@nvidia.com>:
Hi all,
I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create".
When not passing any pg_num and pgp_num values, I get the following error with Quincy:
[root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1
I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed.
I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf:
[root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8
At least, this is what appears to be used with Pacific.
Is this an intended change of behavior? I could not find anything related in the release notes.
Best regards, Geert Kloosterman _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry, hit send too early. It seems I could reproduce it by reducing the value to 1: host1:~ # ceph config set mon mon_max_pool_pg_num 1 host1:~ # ceph config get mon mon_max_pool_pg_num 1 host1:~ # ceph osd pool create pool3 Error ERANGE: 'pg_num' must be greater than 0 and less than or equal to 1 (you may adjust 'mon max pool pg num' for higher values) The default is 65536. Can you verify if this is your issue? Zitat von Eugen Block <eblock@nde.ag>:
Did you ever adjust mon_max_pool_pg_num? Can you check what your current config value is?
host1:~ # ceph config get mon mon_max_pool_pg_num 65536
Zitat von Geert Kloosterman <gkloosterman@nvidia.com>:
Hi,
Thanks Eugen for checking this. I get the same default values as you when I remove the entries from my ceph.conf:
[root@gjk-ceph ~]# ceph-conf -D | grep default_pg osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0
However, in my case, the pool creation error remains:
[root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1
But I can create the pool when passing the same pg_num and pgp_num values explicity:
[root@gjk-ceph ~]# ceph osd pool create asdf 32 0 pool 'asdf' created
Does anyone have an idea how I can debug this further?
I'm running Ceph on a virtualized Rocky 8.7 test cluster, with Ceph rpms installed from http://download.ceph.com/rpm-quincy/el8/
Best regards, Geert Kloosterman
On Wed, 2023-03-15 at 13:42 +0000, Eugen Block wrote:
External email: Use caution opening links or attachments
Hi,
I could not confirm this in a virtual lab cluster, also on 17.2.5:
host1:~ # ceph osd pool create asdf pool 'asdf' created
host1:~ # ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0
So it looks quite similar except the pgp_num value (I can't remember having that modified). This is an upgraded Nautilus cluster.
Zitat von Geert Kloosterman <gkloosterman@nvidia.com>:
Hi all,
I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create".
When not passing any pg_num and pgp_num values, I get the following error with Quincy:
[root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1
I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed.
I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf:
[root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8
At least, this is what appears to be used with Pacific.
Is this an intended change of behavior? I could not find anything related in the release notes.
Best regards, Geert Kloosterman _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Thanks again for your input. The value of mon_max_pool_pg_num was at its default. It turns out I had missed a few steps in my earlier effort: After I removed the old default settings for osd_pool_default_pg_num and osd_pool_default_pgp_num from ceph.conf on *all* nodes, and restarted all ceph services on all nodes, I finally got rid of the error. The error appears to return when setting osd_pool_default_pgp_num to anything else but 0. Even with osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 32 the error returns: Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1 Not sure whether this is intended behavior (to my understanding pg_num and pgp_num should be equal), but at least I can get rid of the error now by not putting these settings in ceph.conf anymore. Cheers, Geert On Tue, 2023-03-21 at 08:03 +0000, Eugen Block wrote:
External email: Use caution opening links or attachments
Sorry, hit send too early. It seems I could reproduce it by reducing the value to 1:
host1:~ # ceph config set mon mon_max_pool_pg_num 1 host1:~ # ceph config get mon mon_max_pool_pg_num 1 host1:~ # ceph osd pool create pool3 Error ERANGE: 'pg_num' must be greater than 0 and less than or equal to 1 (you may adjust 'mon max pool pg num' for higher values)
The default is 65536. Can you verify if this is your issue?
Zitat von Eugen Block <eblock@nde.ag>:
Did you ever adjust mon_max_pool_pg_num? Can you check what your current config value is?
host1:~ # ceph config get mon mon_max_pool_pg_num 65536
Zitat von Geert Kloosterman <gkloosterman@nvidia.com>:
Hi,
Thanks Eugen for checking this. I get the same default values as you when I remove the entries from my ceph.conf:
[root@gjk-ceph ~]# ceph-conf -D | grep default_pg osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0
However, in my case, the pool creation error remains:
[root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1
But I can create the pool when passing the same pg_num and pgp_num values explicity:
[root@gjk-ceph ~]# ceph osd pool create asdf 32 0 pool 'asdf' created
Does anyone have an idea how I can debug this further?
I'm running Ceph on a virtualized Rocky 8.7 test cluster, with Ceph rpms installed from https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdownload.ceph.com%2Frpm-quincy%2Fel8%2F&data=05%7C01%7Cgkloosterman%40nvidia.com%7C7433da6078a54827d52508db29e2fdfb%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638149827139745086%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=a6%2FwrNWvlcb57AW14JVNfJnUI8ryDfAXfirXzuBwl58%3D&reserved=0
Best regards, Geert Kloosterman
On Wed, 2023-03-15 at 13:42 +0000, Eugen Block wrote:
External email: Use caution opening links or attachments
Hi,
I could not confirm this in a virtual lab cluster, also on 17.2.5:
host1:~ # ceph osd pool create asdf pool 'asdf' created
host1:~ # ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 32 osd_pool_default_pgp_num = 0
So it looks quite similar except the pgp_num value (I can't remember having that modified). This is an upgraded Nautilus cluster.
Zitat von Geert Kloosterman <gkloosterman@nvidia.com>:
Hi all,
I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create".
When not passing any pg_num and pgp_num values, I get the following error with Quincy:
[root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1
I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed.
I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf:
[root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8
At least, this is what appears to be used with Pacific.
Is this an intended change of behavior? I could not find anything related in the release notes.
Best regards, Geert Kloosterman _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Geert Kloosterman