MAX AVAIL goes up when I reboot an OSD node
Dear people on this mailing list, I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal. I would love to know WHY. Under normal circumstances I would ignore this behavior, but because I am very new to the whole ceph software I would like to know why stuff like this happens. What I read is, that this value will be calculated by the most filled OSD. I've set noout and norebalance while the node is offline and I unset both values after the reboot. We are currently on nautilus. Cheers and thanks in advance Boris
Is the MGR service colocated on that OSD node? Zitat von Boris Behrens <bb@kervyn.de>:
Dear people on this mailing list,
I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal.
I would love to know WHY.
Under normal circumstances I would ignore this behavior, but because I am very new to the whole ceph software I would like to know why stuff like this happens. What I read is, that this value will be calculated by the most filled OSD.
I've set noout and norebalance while the node is offline and I unset both values after the reboot.
We are currently on nautilus.
Cheers and thanks in advance Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugene, no. The mgr services are located on our mon servers. This happens when I reboot any OSD node. Cheers Boris
Am 29.05.2020 um 08:40 schrieb Eugen Block <eblock@nde.ag>:
Is the MGR service colocated on that OSD node?
Zitat von Boris Behrens <bb@kervyn.de>:
Dear people on this mailing list,
I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal.
I would love to know WHY.
Under normal circumstances I would ignore this behavior, but because I am very new to the whole ceph software I would like to know why stuff like this happens. What I read is, that this value will be calculated by the most filled OSD.
I've set noout and norebalance while the node is offline and I unset both values after the reboot.
We are currently on nautilus.
Cheers and thanks in advance Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Well, this happens when any OSD goes offline. (I stopped a single OSD service on one of our OSD nodes) On Fri, May 29, 2020 at 8:44 AM KervyN <bb@kervyn.de> wrote:
Hi Eugene, no. The mgr services are located on our mon servers.
This happens when I reboot any OSD node.
Zitat von Boris Behrens <bb@kervyn.de>:
Dear people on this mailing list,
I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal.
Does this happen with any random node or specific to 1 node? If specific to 1 node, does this node holds more data compared to other nodes (ceph osd df)? Sinan Polat
Op 29 mei 2020 om 09:56 heeft Boris Behrens <bb@kervyn.de> het volgende geschreven:
Well, this happens when any OSD goes offline. (I stopped a single OSD service on one of our OSD nodes)
On Fri, May 29, 2020 at 8:44 AM KervyN <bb@kervyn.de> wrote:
Hi Eugene, no. The mgr services are located on our mon servers.
This happens when I reboot any OSD node.
Zitat von Boris Behrens <bb@kervyn.de>:
Dear people on this mailing list,
I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Sinan, this happens with any node, and any single OSD. On Fri, May 29, 2020 at 10:09 AM sinan@turka.nl <sinan@turka.nl> wrote:
Does this happen with any random node or specific to 1 node?
If specific to 1 node, does this node holds more data compared to other nodes (ceph osd df)?
Sinan Polat
Does someone got any ideas on this? The mgr nodes are separate, pg_autoscaler is also not active (I don‘t know how the impact will be on a 1pb storage), and it also happens when I turn of an osd service on any node. it’s the latest ceph nautilus. Cheers - Boris
Am 28.05.2020 um 23:42 schrieb Boris Behrens <bb@kervyn.de>:
Dear people on this mailing list,
I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal.
I would love to know WHY.
Under normal circumstances I would ignore this behavior, but because I am very new to the whole ceph software I would like to know why stuff like this happens. What I read is, that this value will be calculated by the most filled OSD.
I've set noout and norebalance while the node is offline and I unset both values after the reboot.
We are currently on nautilus.
Cheers and thanks in advance Boris
I have seen this when there is one OSD on the node being rebooted that is using more space than the others. Max avail for the pool is based on the fullest OSD as far as I know. On Sun, Jun 14, 2020 at 4:29 PM KervyN <bb@kervyn.de> wrote:
Does someone got any ideas on this?
The mgr nodes are separate, pg_autoscaler is also not active (I don‘t know how the impact will be on a 1pb storage), and it also happens when I turn of an osd service on any node.
it’s the latest ceph nautilus.
Cheers - Boris
Am 28.05.2020 um 23:42 schrieb Boris Behrens <bb@kervyn.de>:
Dear people on this mailing list,
I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal.
I would love to know WHY.
Under normal circumstances I would ignore this behavior, but because I am very new to the whole ceph software I would like to know why stuff like this happens. What I read is, that this value will be calculated by the most filled OSD.
I've set noout and norebalance while the node is offline and I unset both values after the reboot.
We are currently on nautilus.
Cheers and thanks in advance Boris
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I restarted all of our osd nodes last week and observed it on all of them. They are also fairly balanced. I have really no idea WHY this happens.
Am 14.06.2020 um 22:36 schrieb Frank R <frankaritchie@gmail.com>:
I have seen this when there is one OSD on the node being rebooted that is using more space than the others. Max avail for the pool is based on the fullest OSD as far as I know.
On Sun, Jun 14, 2020 at 4:29 PM KervyN <bb@kervyn.de> wrote:
Does someone got any ideas on this?
The mgr nodes are separate, pg_autoscaler is also not active (I don‘t know how the impact will be on a 1pb storage), and it also happens when I turn of an osd service on any node.
it’s the latest ceph nautilus.
Cheers - Boris
Am 28.05.2020 um 23:42 schrieb Boris Behrens <bb@kervyn.de>:
Dear people on this mailing list,
I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal.
I would love to know WHY.
Under normal circumstances I would ignore this behavior, but because I am very new to the whole ceph software I would like to know why stuff like this happens. What I read is, that this value will be calculated by the most filled OSD.
I've set noout and norebalance while the node is offline and I unset both values after the reboot.
We are currently on nautilus.
Cheers and thanks in advance Boris
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Exactly my thought too: the node in question has a most-full outlier, so when that OSD is out, the most-full is … less full. ceph osd df | sort -nk8 | tail Or using https://gitlab.cern.ch/ceph/ceph-scripts/blob/e7dbedba66fb585e38f539a41f9640... ceph osd df | egrep -v WEIGHT\|TOTAL\|MIN\|ID\|nan | sed -e 's/ssd//' -e 's/hdd//' | awk '{print 1, $7}' | histogram.py -a -b 200 -m 0 -x 100 -p --no-mvsd (that commandline could be done more elegantly) — Anthony
I have seen this when there is one OSD on the node being rebooted that is using more space than the others. Max avail for the pool is based on the fullest OSD as far as I know.
On Sun, Jun 14, 2020 at 4:29 PM KervyN <bb@kervyn.de> wrote:
Does someone got any ideas on this?
The mgr nodes are separate, pg_autoscaler is also not active (I don‘t know how the impact will be on a 1pb storage), and it also happens when I turn of an osd service on any node.
it’s the latest ceph nautilus.
Cheers - Boris
Am 28.05.2020 um 23:42 schrieb Boris Behrens <bb@kervyn.de>:
Dear people on this mailing list,
I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal.
I would love to know WHY.
Under normal circumstances I would ignore this behavior, but because I am very new to the whole ceph software I would like to know why stuff like this happens. What I read is, that this value will be calculated by the most filled OSD.
I've set noout and norebalance while the node is offline and I unset both values after the reboot.
We are currently on nautilus.
Cheers and thanks in advance Boris
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Anthony D'Atri
-
Boris Behrens
-
Eugen Block
-
Frank R
-
KervyN
-
sinan＠turka.nl