Hi Burkhard, This is a known issue. We ran into it a few months back using VScode containers working on Cephfs under Kubernetes. Tweaking the settings.json file as he suggested here [1] by Dietmar did the trick for us. Regards, Frédéric. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/EUCRAYM3JEG... ----- Le 25 Sep 24, à 9:55, Burkhard Linke Burkhard.Linke@computational.bio.uni-giessen.de a écrit :

Hi,

we are currently trying to debug and understand a problem with cephfs and inotify watchers. A user is running Visual Studio Code with a workspace on a cephfs mount. VSC uses inotify for monitoring files and directories in the workspace:

root@cli:~# ./inotify-info ------------------------------------------------------------------------------ INotify Limits:   max_queued_events    16,384   max_user_instances   128   max_user_watches     1,048,576 ------------------------------------------------------------------------------        Pid Uid        App                       Watches  Instances    3599940 1236       node                        1,681          1          1 0          systemd                       106          5    3600170 1236       node                           54          1     874797 0          udevadm                        17          1    3599118 0          systemd                         7          3    3599707 1236       systemd                         7          3    3599918 1236       node                            6          1       2047 100        dbus-daemon                     3          1       2054 0          sssd                            2          1       2139 0          systemd-logind (deleted)        1          1       2446 0          agetty                          1          1    3600001 1236       node                            1          1 ------------------------------------------------------------------------------ Total inotify Watches:   1886 Total inotify Instances: 20 ------------------------------------------------------------------------------ root@cli:~# cat /sys/kernel/debug/ceph/XYZ.client354064780/caps  | wc -l 1773083

root@cli:~# uname -a Linux cli 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux

So roughly 1.700 watchers result in over 1.7 million caps. (some of the watchers might for files on different filesystems). I've also checked this on the MDS side, it also reports a very high number of caps for that client. Running tools like lsof on the host as root only reports very few open files (<50). So inotify seems to be responsible for the massive caps build up. Terminating VSC results in a sharp drop of the caps  (just a few open files / directories left afterwards).

Is this a known problem?

Best regards,

Burkhard Linke

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io