Re: Cephfs metadta pool suddenly full (100%) !
# ceph status cluster: id: 838506b7-e0c6-4022-9e17-2d1cf9458be6 health: HEALTH_ERR 1 filesystem is degraded 3 full osd(s) 1 pool(s) full 1 daemons have recently crashed You have full OSDs and therefore a full pool. The "fullness" of a pool is limited by the fullest OSD, i.e. a single full OSDs can block your
Hi Hervé, On 01.06.21 13:15, Hervé Ballans wrote: pool. Take a look at `ceph osd df` and you will notice a very non-uniform osd usage (both with numbers of PG / size as well as usage %).
osd: 126 osds: 126 up (since 5m), 126 in (since 5M) pgs: 1662 active+clean
The PG/osd ratio seems to be very low for me. The general recommendation is 100 PG / osd post-replication (and power of 2 for each pool). In my cluster I actually run with ~200 PG / osd for my SSD which contain the cephfs metadata.
Thanks a lot if you have some ways for trying to solve this...
You have to get your OSDs to rebalance, which probably includes increasing the number of PGs in some pools. Details depend on which Ceph version you are running and your CRUSH rules (maybe your cephfs metadata pool is residing only on NVMe?). Take a look at the balancer module [1] and the autoscaler [2] (`ceph osd pool autoscale-status` is most interesting). Theoretically, you could (temporarilly!) increase the full_ratio. However, this is a very dangerous operation which you should not do unless you know *exactly* what you are doing. Cheers & Best of luck Sebastian [1] https://docs.ceph.com/en/latest/rados/operations/balancer/ [2] https://docs.ceph.com/en/latest/rados/operations/placement-groups/ Replace latest in the URIs with your Ceph version string (i.e. octopus, nautilus) for version specific documentation
Hi Sebastian, Thank you for your quick answer. I'm aware with your points, and maybe I was not really clear in my previous email (written in a hurry!) The problematic pool is the metadata one. All its OSDs (x3) are full. The associated data pool is OK and no OSD is full on the data pool. The problem is that metadata pool suddenly increases a lot and continiously from 3% to 100% in 5 hours (from 5 am to 10 am, then crash) And we don't understand the reason, since there was no specific activities on the data pool ? This cluster runs perfectly with the current configuration since many years. Thanks again, Hervé Le 01/06/2021 à 13:42, Sebastian Knust a écrit :
Hi Hervé,
# ceph status cluster: id: 838506b7-e0c6-4022-9e17-2d1cf9458be6 health: HEALTH_ERR 1 filesystem is degraded 3 full osd(s) 1 pool(s) full 1 daemons have recently crashed You have full OSDs and therefore a full pool. The "fullness" of a pool is limited by the fullest OSD, i.e. a single full OSDs can block your
On 01.06.21 13:15, Hervé Ballans wrote: pool. Take a look at `ceph osd df` and you will notice a very non-uniform osd usage (both with numbers of PG / size as well as usage %).
osd: 126 osds: 126 up (since 5m), 126 in (since 5M) pgs: 1662 active+clean
The PG/osd ratio seems to be very low for me. The general recommendation is 100 PG / osd post-replication (and power of 2 for each pool). In my cluster I actually run with ~200 PG / osd for my SSD which contain the cephfs metadata.
Thanks a lot if you have some ways for trying to solve this...
You have to get your OSDs to rebalance, which probably includes increasing the number of PGs in some pools. Details depend on which Ceph version you are running and your CRUSH rules (maybe your cephfs metadata pool is residing only on NVMe?). Take a look at the balancer module [1] and the autoscaler [2] (`ceph osd pool autoscale-status` is most interesting).
Theoretically, you could (temporarilly!) increase the full_ratio. However, this is a very dangerous operation which you should not do unless you know *exactly* what you are doing.
Cheers & Best of luck Sebastian
[1] https://docs.ceph.com/en/latest/rados/operations/balancer/ [2] https://docs.ceph.com/en/latest/rados/operations/placement-groups/ Replace latest in the URIs with your Ceph version string (i.e. octopus, nautilus) for version specific documentation
participants (2)
-
Hervé Ballans
-
Sebastian Knust