Hi Guys, I've got a funny one I'm hoping someone can point me in the right direction with: We've got three identical(?) Ceph nodes running 4 OSDs, Mon, Man, and iSCSI G/W each (we're only a small shop) on Rocky Linux 8 / Ceph Quincy. Everything is running fine, no bottle-necks (as far as we can see) and the Cluster is holding up very well. However, one of the boxes is constantly running out of space on the /var mount. Its 16 GiB in size, and it only takes a day or three to fill up, thus taking it's monitor service out of quorum. The thing is, I can't find *what's* taking up all the space. At first we thought it was an overly large log file, but I've done searches to find the largest files, etc, and nothing is showing up (that I can find) - ie the log files on this box are comparable with the log files on the other two boxes and the other two boxes are sitting at around 10% full (via a df-H), while the problem box is at around 85% and growing (at time of posting). Another interesting point is that the problem box, unrelated to this issue, was rebooted recently and when it came back on-line the space-issue was gone ie the /var mount was back down to around the 10% mark. This suggests to me its some sort of "temporary" journal/log/dump/whatever/? that was "reset" (cleaned-up?) via the reboot. I've had a look at the logs but I'm not sure what I should be looking for - so I don't even know if I'm looking in the *correct* logs... Anyone got any ideas? I mean, rebooting the server every couple of days is not really a practical solution, and neither is turning off the monitor service on the box, and increasing the size of the /var mount just seems like it'll postpone the issue. Any help would be greatly appreciated. Cheers Dulux-Oz
Hi All, Thanks to Eneko Lacunza, E Taka, and Anthony D'Atri for replying - all that advice was really helpful. So, we finally tracked down our "disk eating monster" (sort of). We've got a "runaway" ceph-guest-NN that is filling up its log file (/var/log/ceph/ceph-guest-NN.log) and eventually over-flowing the /var partition. What we haven't been able to do yet is actually track-down the "ceph-guest-NN" process so we can kill it. Restarting the monitor service on that node "pauses" the offending process, but as soon the mon service restarts the relevant log file is re-created/begins to fill up again. What we've done in the mean time is put cron job in place to run once a day to delete the offending log file - that's keeping us online, but it's a less-than-optimal solution (ie it's a "hack"). Soooooo... anyone got any pointers as to how we can go about actually finding the offending process? Cheers Dulux-Oz
Hi, El 17/1/23 a las 8:12, duluxoz escribió:
Thanks to Eneko Lacunza, E Taka, and Anthony D'Atri for replying - all that advice was really helpful.
So, we finally tracked down our "disk eating monster" (sort of). We've got a "runaway" ceph-guest-NN that is filling up its log file (/var/log/ceph/ceph-guest-NN.log) and eventually over-flowing the /var partition.
What we haven't been able to do yet is actually track-down the "ceph-guest-NN" process so we can kill it. Restarting the monitor service on that node "pauses" the offending process, but as soon the mon service restarts the relevant log file is re-created/begins to fill up again. What we've done in the mean time is put cron job in place to run once a day to delete the offending log file - that's keeping us online, but it's a less-than-optimal solution (ie it's a "hack").
Soooooo... anyone got any pointers as to how we can go about actually finding the offending process?
Do you really need that log file? What about disabling it? I have never seen such a log file, I think it could be a debug log defined in ceph.conf in some client section with "log file"... Cheers Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 |https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
Hi Eneko, Well, that's the thing: there are a whole bunch of ceph-guest-XX.log files in /var/log/caeh/; most of them are empty, a handful are up to 250 Kb in size, and this one () keeps on growing - and where not sure where they're coming from (ie there's nothing that we can see in the conf files. However, your "debug" comment has sparked a vague memory, so I'll check that out tomorrow (its after dinner here in Aus, so I'll tackle it first thing tomorrow - now that we've got our "hack" in place and we don't have to worry about the cluster in the short term :-) Cheers Dulux-Oz
Well, that's the thing: there are a whole bunch of ceph-guest-XX.log files in /var/log/ceph/; most of them are empty, a handful are up to 250 Kb in size, and this one () keeps on growing - and where not sure where they're coming from (ie there's nothing that we can see in the conf files.
The program "lsof" should be able to tell you which PID hold the file descriptor for the file when it is growing, that would probably help narrow it down. -- May the most significant bit of your life be positive.
participants (4)
-
duluxoz
-
Eneko Lacunza
-
Janne Johansson
-
matthew＠peregrineit.net