11 Jan
2023
11 Jan
'23
8:16 a.m.
Hi Guys,
I've got a funny one I'm hoping someone can point me in the right direction with:
We've got three identical(?) Ceph nodes running 4 OSDs, Mon, Man, and iSCSI G/W each
(we're only a small shop) on Rocky Linux 8 / Ceph Quincy. Everything is running fine,
no bottle-necks (as far as we can see) and the Cluster is holding up very well.
However, one of the boxes is constantly running out of space on the /var mount. Its 16 GiB
in size, and it only takes a day or three to fill up, thus taking it's monitor service
out of quorum.
The thing is, I can't find *what's* taking up all the space. At first we thought
it was an overly large log file, but I've done searches to find the largest files,
etc, and nothing is showing up (that I can find) - ie the log files on this box are
comparable with the log files on the other two boxes and the other two boxes are sitting
at around 10% full (via a df-H), while the problem box is at around 85% and growing (at
time of posting).
Another interesting point is that the problem box, unrelated to this issue, was rebooted
recently and when it came back on-line the space-issue was gone ie the /var mount was back
down to around the 10% mark.
This suggests to me its some sort of "temporary" journal/log/dump/whatever/?
that was "reset" (cleaned-up?) via the reboot.
I've had a look at the logs but I'm not sure what I should be looking for - so I
don't even know if I'm looking in the *correct* logs...
Anyone got any ideas? I mean, rebooting the server every couple of days is not really a
practical solution, and neither is turning off the monitor service on the box, and
increasing the size of the /var mount just seems like it'll postpone the issue.
Any help would be greatly appreciated.
Cheers
Dulux-Oz
17 Jan
17 Jan
10:42 a.m.
Hi All,
Thanks to Eneko Lacunza, E Taka, and Anthony D'Atri for replying - all
that advice was really helpful.
So, we finally tracked down our "disk eating monster" (sort of). We've
got a "runaway" ceph-guest-NN that is filling up its log file
(/var/log/ceph/ceph-guest-NN.log) and eventually over-flowing the /var
partition.
What we haven't been able to do yet is actually track-down the
"ceph-guest-NN" process so we can kill it. Restarting the monitor
service on that node "pauses" the offending process, but as soon the mon
service restarts the relevant log file is re-created/begins to fill up
again. What we've done in the mean time is put cron job in place to run
once a day to delete the offending log file - that's keeping us online,
but it's a less-than-optimal solution (ie it's a "hack").
Soooooo... anyone got any pointers as to how we can go about actually
finding the offending process?
Cheers
Dulux-Oz
12:05 p.m.
Hi,
El 17/1/23 a las 8:12, duluxoz escribió:
Thanks to Eneko Lacunza, E Taka, and Anthony D'Atri for replying - all
that advice was really helpful.
So, we finally tracked down our "disk eating monster" (sort of). We've
got a "runaway" ceph-guest-NN that is filling up its log file
(/var/log/ceph/ceph-guest-NN.log) and eventually over-flowing the /var
partition.
What we haven't been able to do yet is actually track-down the
"ceph-guest-NN" process so we can kill it. Restarting the monitor
service on that node "pauses" the offending process, but as soon the
mon service restarts the relevant log file is re-created/begins to
fill up again. What we've done in the mean time is put cron job in
place to run once a day to delete the offending log file - that's
keeping us online, but it's a less-than-optimal solution (ie it's a
"hack").
Soooooo... anyone got any pointers as to how we can go about actually
finding the offending process?
Do you really need that log file? What about disabling it? I have never
seen such a log file, I think it could be a debug log defined in
ceph.conf in some client section with "log file"...
Cheers
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
12:20 p.m.
Hi Eneko,
Well, that's the thing: there are a whole bunch of ceph-guest-XX.log
files in /var/log/caeh/; most of them are empty, a handful are up to 250
Kb in size, and this one () keeps on growing - and where not sure where
they're coming from (ie there's nothing that we can see in the conf files.
However, your "debug" comment has sparked a vague memory, so I'll check
that out tomorrow (its after dinner here in Aus, so I'll tackle it first
thing tomorrow - now that we've got our "hack" in place and we don't
have to worry about the cluster in the short term :-)
Cheers
Dulux-Oz
5:08 p.m.
Well, that's the thing: there are a whole bunch of
ceph-guest-XX.log
files in /var/log/ceph/; most of them are empty, a handful are up to 250
Kb in size, and this one () keeps on growing - and where not sure where
they're coming from (ie there's nothing that we can see in the conf files.
The program "lsof" should be able to tell you which PID hold the file
descriptor for the file when it is growing, that would probably help
narrow it down.
--
May the most significant bit of your life be positive.
489
days inactive
495
days old
4 comments
4 participants
participants (4)
-
duluxoz -
Eneko Lacunza -
Janne Johansson -
matthew＠peregrineit.net