Mysterious Disk-Space Eater
Hi All, Got a funny one, which I'm hoping someone can help us with. We've got three identical(?) Ceph Quincy Nodes running on Rocky Linux 8.7. Each Node has 4 OSDs, plus Monitor, Manager, and iSCSI G/W services running on them (we're only a small shop). Each Node has a separate 16 GiB partition mounted as /var. Everything is running well and the Ceph Cluster is handling things very well). However, one of the Nodes (not the one currently acting as the Active Manager) is running out of space on /var. Normally, all of the Nodes have around 10% space used (via a df -H command), but the problem Node only takes 1 to 3 days to run out of space, hence taking it out of Quorum. Its currently at 85% and growing. At first we thought this was caused by an overly large log file, but investigations showed that all the logs on all 3 Nodes were of comparable size. Also, searching for the 20 largest files on the problem Node's /var didn't produce any significant results. Coincidentally, unrelated to this issue, the problem Node (but not the other 2 Nodes) was re-booted a couple of days ago and, when the Cluster had re-balanced itself and everything was back online and reporting as Healthy, the problem Node's /var was back down to around 10%, the same as the other two Nodes. This lead us to suspect that there was some sort of "run-away" process or journaling/logging/temporary file(s) or whatever that the re-boot has "cleaned up". So we've been keeping an eye on things but we can't see anything causing the issue and now, as I said above, the problem Node's /var is back up to 85% and growing. I've been looking at the log files, tying to determine the issue, but as I don't really know what I'm looking for I don't even know if I'm looking in the *correct* log files... Obviously rebooting the problem Node every couple of days is not a viable option, and increasing the size of the /var partition is only going to postpone the issue, not resolve it. So if anyone has any ideas we'd love to hear about it - thanks Cheers Dulux-Oz
Hi, El 12/1/23 a las 3:59, duluxoz escribió:
Got a funny one, which I'm hoping someone can help us with.
We've got three identical(?) Ceph Quincy Nodes running on Rocky Linux 8.7. Each Node has 4 OSDs, plus Monitor, Manager, and iSCSI G/W services running on them (we're only a small shop). Each Node has a separate 16 GiB partition mounted as /var. Everything is running well and the Ceph Cluster is handling things very well).
However, one of the Nodes (not the one currently acting as the Active Manager) is running out of space on /var. Normally, all of the Nodes have around 10% space used (via a df -H command), but the problem Node only takes 1 to 3 days to run out of space, hence taking it out of Quorum. Its currently at 85% and growing.
At first we thought this was caused by an overly large log file, but investigations showed that all the logs on all 3 Nodes were of comparable size. Also, searching for the 20 largest files on the problem Node's /var didn't produce any significant results.
Coincidentally, unrelated to this issue, the problem Node (but not the other 2 Nodes) was re-booted a couple of days ago and, when the Cluster had re-balanced itself and everything was back online and reporting as Healthy, the problem Node's /var was back down to around 10%, the same as the other two Nodes.
This lead us to suspect that there was some sort of "run-away" process or journaling/logging/temporary file(s) or whatever that the re-boot has "cleaned up". So we've been keeping an eye on things but we can't see anything causing the issue and now, as I said above, the problem Node's /var is back up to 85% and growing.
I've been looking at the log files, tying to determine the issue, but as I don't really know what I'm looking for I don't even know if I'm looking in the *correct* log files...
Obviously rebooting the problem Node every couple of days is not a viable option, and increasing the size of the /var partition is only going to postpone the issue, not resolve it. So if anyone has any ideas we'd love to hear about it - thanks
This seems one or more files that are removed but some process has their handle open (and maybe is still writing...). When rebooting process is terminated and file(s) effectively removed. Try to inspect each process' open files and find what file(s) have no longer a directory entry... that would give you a hint. Cheers Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
We had a similar problem, and it was a (visible) logfile. It is easy to find with the ncdu utility (`ncdu -x /var`). There's no need of a reboot, you can get rid of it with restarting the Monitor with `ceph orch daemon restart mon.NODENAME`. You may also lower the debug level. Am Do., 12. Jan. 2023 um 09:14 Uhr schrieb Eneko Lacunza <elacunza@binovo.es
:
Hi,
El 12/1/23 a las 3:59, duluxoz escribió:
Got a funny one, which I'm hoping someone can help us with.
We've got three identical(?) Ceph Quincy Nodes running on Rocky Linux 8.7. Each Node has 4 OSDs, plus Monitor, Manager, and iSCSI G/W services running on them (we're only a small shop). Each Node has a separate 16 GiB partition mounted as /var. Everything is running well and the Ceph Cluster is handling things very well).
However, one of the Nodes (not the one currently acting as the Active Manager) is running out of space on /var. Normally, all of the Nodes have around 10% space used (via a df -H command), but the problem Node only takes 1 to 3 days to run out of space, hence taking it out of Quorum. Its currently at 85% and growing.
At first we thought this was caused by an overly large log file, but investigations showed that all the logs on all 3 Nodes were of comparable size. Also, searching for the 20 largest files on the problem Node's /var didn't produce any significant results.
Coincidentally, unrelated to this issue, the problem Node (but not the other 2 Nodes) was re-booted a couple of days ago and, when the Cluster had re-balanced itself and everything was back online and reporting as Healthy, the problem Node's /var was back down to around 10%, the same as the other two Nodes.
This lead us to suspect that there was some sort of "run-away" process or journaling/logging/temporary file(s) or whatever that the re-boot has "cleaned up". So we've been keeping an eye on things but we can't see anything causing the issue and now, as I said above, the problem Node's /var is back up to 85% and growing.
I've been looking at the log files, tying to determine the issue, but as I don't really know what I'm looking for I don't even know if I'm looking in the *correct* log files...
Obviously rebooting the problem Node every couple of days is not a viable option, and increasing the size of the /var partition is only going to postpone the issue, not resolve it. So if anyone has any ideas we'd love to hear about it - thanks
This seems one or more files that are removed but some process has their handle open (and maybe is still writing...). When rebooting process is terminated and file(s) effectively removed.
Try to inspect each process' open files and find what file(s) have no longer a directory entry... that would give you a hint.
Cheers
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
One can even remove the log and tell the daemon to reopen it without having to restart. I’ve had mons do enough weird things on me that I try to avoid restarting them. ymmv. It’s possible that the OP has a large file that’s unlinked but still open, historically “fsck -n” would find these, today that would depend on the filesystem in use. It’s also possible that there is data under a mountpoint directory within /var, that’s masked by the overlaid mount. http://cephnotes.ksperis.com/blog/2017/01/20/change-log-level-on-the-fly-to-ceph-daemons/
On Jan 12, 2023, at 4:04 AM, E Taka <0etaka0@gmail.com> wrote:
We had a similar problem, and it was a (visible) logfile. It is easy to find with the ncdu utility (`ncdu -x /var`). There's no need of a reboot, you can get rid of it with restarting the Monitor with `ceph orch daemon restart mon.NODENAME`. You may also lower the debug level.
Am Do., 12. Jan. 2023 um 09:14 Uhr schrieb Eneko Lacunza <elacunza@binovo.es
:
Hi,
El 12/1/23 a las 3:59, duluxoz escribió:
Got a funny one, which I'm hoping someone can help us with.
We've got three identical(?) Ceph Quincy Nodes running on Rocky Linux 8.7. Each Node has 4 OSDs, plus Monitor, Manager, and iSCSI G/W services running on them (we're only a small shop). Each Node has a separate 16 GiB partition mounted as /var. Everything is running well and the Ceph Cluster is handling things very well).
However, one of the Nodes (not the one currently acting as the Active Manager) is running out of space on /var. Normally, all of the Nodes have around 10% space used (via a df -H command), but the problem Node only takes 1 to 3 days to run out of space, hence taking it out of Quorum. Its currently at 85% and growing.
At first we thought this was caused by an overly large log file, but investigations showed that all the logs on all 3 Nodes were of comparable size. Also, searching for the 20 largest files on the problem Node's /var didn't produce any significant results.
Coincidentally, unrelated to this issue, the problem Node (but not the other 2 Nodes) was re-booted a couple of days ago and, when the Cluster had re-balanced itself and everything was back online and reporting as Healthy, the problem Node's /var was back down to around 10%, the same as the other two Nodes.
This lead us to suspect that there was some sort of "run-away" process or journaling/logging/temporary file(s) or whatever that the re-boot has "cleaned up". So we've been keeping an eye on things but we can't see anything causing the issue and now, as I said above, the problem Node's /var is back up to 85% and growing.
I've been looking at the log files, tying to determine the issue, but as I don't really know what I'm looking for I don't even know if I'm looking in the *correct* log files...
Obviously rebooting the problem Node every couple of days is not a viable option, and increasing the size of the /var partition is only going to postpone the issue, not resolve it. So if anyone has any ideas we'd love to hear about it - thanks
This seems one or more files that are removed but some process has their handle open (and maybe is still writing...). When rebooting process is terminated and file(s) effectively removed.
Try to inspect each process' open files and find what file(s) have no longer a directory entry... that would give you a hint.
Cheers
Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
duluxoz
-
E Taka
-
Eneko Lacunza